2024-09-04T21:40:02.073 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-04T21:40:02.090 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-04T21:40:02.173 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472 branch: wip-dis-testing description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache} email: idryomov@gmail.com first_in_suite: false job_id: '7889472' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: rbd cache: false global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: live-migration-nbd sha1: 250c2b7303ac79643ee967bc9c5b14e5a0dbbf52 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6930 sha1: 3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab sleep_before_teardown: 0 subset: 4/100 suite: rbd suite_branch: live-migration-nbd suite_path: /home/teuthworker/src/github.com_effi-ofer_ceph_250c2b7303ac79643ee967bc9c5b14e5a0dbbf52/qa suite_relpath: qa suite_repo: https://github.com/effi-ofer/ceph suite_sha1: 250c2b7303ac79643ee967bc9c5b14e5a0dbbf52 targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFoBeeVfhMhHR6V9cArsNl6U3ls8t74E+zhkX0tWvC78u1HGP0Ot1Q671DhxNoswYHbH9sUdt/jiyog/NppFqg= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBORv0KlzsNxY52E2puIKF/wdLgc8vSAwRXY36mQ8K6yQR9oeqsX6NMIlzLz2D7Hc5wjxM90wWQ0pI3q6pssoEek= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-04_17:03:08 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-04T21:40:02.174 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_effi-ofer_ceph_250c2b7303ac79643ee967bc9c5b14e5a0dbbf52/qa; will attempt to use it 2024-09-04T21:40:02.174 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_effi-ofer_ceph_250c2b7303ac79643ee967bc9c5b14e5a0dbbf52/qa/tasks 2024-09-04T21:40:02.174 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-04T21:40:02.176 INFO:teuthology.task.internal:Checking packages... 2024-09-04T21:40:02.198 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab' 2024-09-04T21:40:02.198 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-04T21:40:02.198 INFO:teuthology.packaging:ref: None 2024-09-04T21:40:02.198 INFO:teuthology.packaging:tag: None 2024-09-04T21:40:02.198 INFO:teuthology.packaging:branch: wip-dis-testing 2024-09-04T21:40:02.198 INFO:teuthology.packaging:sha1: 3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab 2024-09-04T21:40:02.198 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2024-09-04T21:40:02.362 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:40:02.363 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-04T21:40:02.370 INFO:teuthology.task.internal:no buildpackages task found 2024-09-04T21:40:02.370 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-04T21:40:02.409 INFO:teuthology.task.internal:Saving configuration 2024-09-04T21:40:02.421 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-04T21:40:02.436 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-04T21:40:02.460 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-04 21:33:52.643721', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM05whNtWmV5LgFAP2nRBAeIQzlpoJkOR2xVYsVWBa7U'} 2024-09-04T21:40:02.482 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-04 21:33:52.645289', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBHeySmp7qg90lBANx/zpov9HEPx+HxViOUNzdArlcJh'} 2024-09-04T21:40:02.482 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-04T21:40:02.499 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-04T21:40:02.500 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-04T21:40:02.500 INFO:teuthology.run_tasks:Running task console_log... 2024-09-04T21:40:02.604 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c696cfac0>, signals=[15]) 2024-09-04T21:40:02.604 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-04T21:40:02.611 INFO:teuthology.task.internal:Opening connections... 2024-09-04T21:40:02.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:40:02.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:40:02.709 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:40:02.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:40:02.789 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-04T21:40:02.796 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-09-04T21:40:02.801 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-09-04T21:40:02.801 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-09-04T21:40:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-04T21:40:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2024-09-04T21:40:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="22.04" 2024-09-04T21:40:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-04T21:40:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=jammy 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-04T21:40:02.846 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=jammy 2024-09-04T21:40:02.847 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-09-04T21:40:02.876 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-09-04T21:40:02.882 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-09-04T21:40:02.883 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-09-04T21:40:02.933 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-04T21:40:02.933 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Ubuntu" 2024-09-04T21:40:02.933 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="22.04" 2024-09-04T21:40:02.934 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-04T21:40:02.934 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_CODENAME=jammy 2024-09-04T21:40:02.934 INFO:teuthology.orchestra.run.smithi157.stdout:ID=ubuntu 2024-09-04T21:40:02.934 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE=debian 2024-09-04T21:40:02.934 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-04T21:40:02.934 INFO:teuthology.orchestra.run.smithi157.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-04T21:40:02.935 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-04T21:40:02.935 INFO:teuthology.orchestra.run.smithi157.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-04T21:40:02.935 INFO:teuthology.orchestra.run.smithi157.stdout:UBUNTU_CODENAME=jammy 2024-09-04T21:40:02.935 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-09-04T21:40:02.967 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-04T21:40:02.980 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-04T21:40:03.031 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-04T21:40:03.032 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-09-04T21:40:03.035 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-09-04T21:40:03.039 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-04T21:40:03.068 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-04T21:40:03.068 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-09-04T21:40:03.083 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-09-04T21:40:03.098 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-04T21:40:03.152 INFO:teuthology.run_tasks:Running task kernel... 2024-09-04T21:40:03.168 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-04T21:40:03.168 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-04T21:40:03.168 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-04T21:40:03.169 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-04T21:40:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:40:03.174 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-04T21:40:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:40:03.175 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-04T21:40:03.223 INFO:teuthology.orchestra.run.smithi112.stdout:5.15.0-56-generic 2024-09-04T21:40:03.223 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.15.0-56-generic 2024-09-04T21:40:03.223 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-04T21:40:03.225 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-56-generic 2024-09-04T21:40:03.225 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.15.0-56-generic 2024-09-04T21:40:03.226 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2024-09-04T21:40:03.354 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-04T21:40:03.360 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-09-04T21:40:03.498 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-04T21:40:03.606 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T21:40:03.656 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T21:40:03.656 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-04T21:40:03.688 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-04T21:40:03.703 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-04T21:40:03.714 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-04T21:40:03.804 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-04T21:40:03.844 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-04T21:40:03.850 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-04T21:40:03.851 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-04T21:40:03.884 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-04T21:40:03.884 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-04T21:40:03.889 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-04T21:40:03.889 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-04T21:40:03.895 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-04T21:40:03.902 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-04T21:40:03.904 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-04T21:40:03.904 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-04T21:40:03.904 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-04T21:40:03.985 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-04T21:40:03.985 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-04T21:40:04.044 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-04T21:40:04.058 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-04T21:40:04.080 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-04T21:40:04.182 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-04T21:40:04.196 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-04T21:40:04.376 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-04T21:40:04.400 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-04T21:40:04.500 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-04T21:40:04.516 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-04T21:40:04.525 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-04T21:40:04.525 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-04T21:40:04.526 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-04T21:40:04.533 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-04T21:40:04.547 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-04T21:40:04.548 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-04T21:40:04.548 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-04T21:40:04.549 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-04T21:40:04.562 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-04T21:40:04.562 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-04T21:40:04.564 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-04T21:40:04.616 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-04T21:40:04.621 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-04T21:40:04.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-04T21:40:04.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-04T21:40:04.664 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-04T21:40:04.684 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-04T21:40:04.684 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-04T21:40:04.685 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-04T21:40:04.685 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-04T21:40:04.689 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-04T21:40:04.692 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-04T21:40:04.692 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-04T21:40:04.693 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-04T21:40:04.696 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-04T21:40:04.697 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-04T21:40:04.697 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-04T21:40:04.700 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-04T21:40:04.702 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-04T21:40:04.703 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-04T21:40:04.703 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-04T21:40:04.703 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-04T21:40:04.705 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-04T21:40:04.724 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-04T21:40:04.728 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-04T21:40:04.729 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-04T21:40:04.729 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-04T21:40:04.730 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-04T21:40:04.761 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-04T21:40:04.771 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-04T21:40:04.771 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-04T21:40:04.772 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-04T21:40:04.773 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-04T21:40:04.773 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-04T21:40:04.778 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-04T21:40:04.778 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-04T21:40:04.793 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-04T21:40:04.793 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-04T21:40:04.794 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-04T21:40:04.795 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-04T21:40:04.859 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-04T21:40:04.860 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-04T21:40:04.874 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-04T21:40:04.874 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-04T21:40:04.875 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-04T21:40:04.875 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-04T21:40:04.876 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-04T21:40:10.234 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 15.8 MB in 2s (7,375 kB/s) 2024-09-04T21:40:10.848 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 15.8 MB in 3s (5,789 kB/s) 2024-09-04T21:40:11.710 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-04T21:40:11.728 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-04T21:40:11.793 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-04T21:40:12.003 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-09-04T21:40:12.003 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-09-04T21:40:12.179 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T21:40:12.179 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T21:40:12.180 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T21:40:12.180 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T21:40:12.180 INFO:teuthology.orchestra.run.smithi157.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-04T21:40:12.180 INFO:teuthology.orchestra.run.smithi157.stdout: python2.7-minimal 2024-09-04T21:40:12.180 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T21:40:12.181 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-09-04T21:40:12.181 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-04T21:40:12.181 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-04T21:40:12.182 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-04T21:40:12.183 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-09-04T21:40:12.183 INFO:teuthology.orchestra.run.smithi157.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-04T21:40:12.183 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2024-09-04T21:40:12.183 INFO:teuthology.orchestra.run.smithi157.stdout: thermald 2024-09-04T21:40:12.226 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-09-04T21:40:12.227 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-04T21:40:12.227 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-04T21:40:12.227 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-04T21:40:12.228 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages will be upgraded: 2024-09-04T21:40:12.229 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-04T21:40:12.293 INFO:teuthology.orchestra.run.smithi157.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-04T21:40:12.293 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 113 MB of archives. 2024-09-04T21:40:12.293 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-04T21:40:12.293 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-04T21:40:12.369 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-04T21:40:12.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-04T21:40:12.449 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-04T21:40:12.637 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-04T21:40:12.637 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-04T21:40:12.737 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-04T21:40:12.805 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T21:40:12.805 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T21:40:12.805 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T21:40:12.805 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T21:40:12.806 INFO:teuthology.orchestra.run.smithi112.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-04T21:40:12.806 INFO:teuthology.orchestra.run.smithi112.stdout: python2.7-minimal 2024-09-04T21:40:12.806 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T21:40:12.807 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-04T21:40:12.807 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-04T21:40:12.807 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-04T21:40:12.807 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-04T21:40:12.808 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-04T21:40:12.808 INFO:teuthology.orchestra.run.smithi112.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-04T21:40:12.809 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-04T21:40:12.809 INFO:teuthology.orchestra.run.smithi112.stdout: thermald 2024-09-04T21:40:12.856 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-04T21:40:12.857 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-04T21:40:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-04T21:40:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-04T21:40:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-04T21:40:12.859 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2024-09-04T21:40:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-04T21:40:12.910 INFO:teuthology.orchestra.run.smithi112.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-04T21:40:12.911 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 113 MB of archives. 2024-09-04T21:40:12.911 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-04T21:40:12.911 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-04T21:40:13.333 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-04T21:40:13.438 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-04T21:40:13.692 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-04T21:40:13.692 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-04T21:40:13.692 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-04T21:40:13.820 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-04T21:40:13.843 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-04T21:40:14.231 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 113 MB in 2s (70.5 MB/s) 2024-09-04T21:40:14.241 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-04T21:40:14.241 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-04T21:40:14.241 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-04T21:40:14.365 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-04T21:40:14.439 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-04T21:40:14.465 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-04T21:40:14.864 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 113 MB in 2s (70.9 MB/s) 2024-09-04T21:40:15.071 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-04T21:40:16.428 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-04T21:40:16.432 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:16.528 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:17.124 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-04T21:40:17.128 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:17.361 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:18.780 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-04T21:40:18.801 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:18.901 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:19.358 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-04T21:40:19.379 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:19.416 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:19.604 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-04T21:40:19.625 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:19.779 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:20.516 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-04T21:40:20.536 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:20.574 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:26.592 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T21:40:26.721 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T21:40:27.043 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T21:40:27.139 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T21:40:27.445 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T21:40:27.476 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-04T21:40:27.498 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-04T21:40:27.542 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-04T21:40:27.560 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T21:40:27.909 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T21:40:28.006 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T21:40:28.325 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-04T21:40:28.348 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-04T21:40:28.392 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-04T21:40:33.613 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-04T21:40:33.651 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:33.688 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:34.282 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-04T21:40:34.318 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T21:40:34.348 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:36.367 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T21:40:36.489 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T21:40:36.823 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-04T21:40:36.961 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:37.087 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-04T21:40:37.216 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:37.711 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T21:40:37.826 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T21:40:38.201 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-04T21:40:38.356 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:38.482 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-04T21:40:38.529 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-04T21:40:38.529 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-04T21:40:38.529 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-04T21:40:38.529 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-04T21:40:38.636 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:38.717 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:39.961 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-04T21:40:39.961 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-04T21:40:39.961 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-04T21:40:39.961 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-04T21:40:40.154 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:40.327 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:41.630 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:41.694 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-04T21:40:41.854 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-04T21:40:41.980 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:42.054 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-04T21:40:42.054 INFO:teuthology.orchestra.run.smithi157.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-04T21:40:43.040 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-04T21:40:43.199 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-04T21:40:43.325 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T21:40:43.414 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-04T21:40:43.414 INFO:teuthology.orchestra.run.smithi112.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-04T21:40:56.255 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-04T21:40:56.255 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub' 2024-09-04T21:40:56.286 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-04T21:40:56.314 INFO:teuthology.orchestra.run.smithi157.stdout:Generating grub configuration file ... 2024-09-04T21:40:56.607 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-04T21:40:56.632 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-04T21:40:56.944 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-04T21:40:56.950 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-04T21:40:57.023 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-04T21:40:57.028 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-04T21:40:57.202 INFO:teuthology.orchestra.run.smithi157.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-04T21:40:57.202 INFO:teuthology.orchestra.run.smithi157.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-04T21:40:57.202 INFO:teuthology.orchestra.run.smithi157.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-04T21:40:57.230 INFO:teuthology.orchestra.run.smithi157.stdout:done 2024-09-04T21:40:57.646 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2024-09-04T21:40:57.663 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2024-09-04T21:40:57.663 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 21 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.119.119 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2024-09-04T21:40:57.664 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2024-09-04T21:40:57.665 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-04T21:40:57.665 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-04T21:40:57.665 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-04T21:40:57.665 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-04T21:40:57.665 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-09-04T21:40:57.665 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2024-09-04T21:40:57.678 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-04T21:40:57.678 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub' 2024-09-04T21:40:57.703 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-04T21:40:57.729 INFO:teuthology.orchestra.run.smithi112.stdout:Generating grub configuration file ... 2024-09-04T21:40:57.828 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-09-04T21:40:57.946 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T21:40:57.956 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-04T21:40:57.973 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-04T21:40:58.031 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-04T21:40:58.056 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-04T21:40:58.082 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-04T21:40:58.426 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-04T21:40:58.432 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-04T21:40:58.499 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-04T21:40:58.505 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-04T21:40:58.655 INFO:teuthology.orchestra.run.smithi112.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-04T21:40:58.655 INFO:teuthology.orchestra.run.smithi112.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-04T21:40:58.655 INFO:teuthology.orchestra.run.smithi112.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-04T21:40:58.703 INFO:teuthology.orchestra.run.smithi112.stdout:done 2024-09-04T21:40:59.191 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-04T21:40:59.223 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-04T21:40:59.223 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 21 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.15.0.119.119 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-04T21:40:59.224 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-04T21:40:59.225 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-04T21:40:59.225 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-04T21:40:59.226 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-04T21:40:59.226 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-04T21:40:59.226 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-09-04T21:40:59.226 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-04T21:40:59.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-04T21:40:59.443 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-04T21:40:59.461 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-04T21:40:59.515 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-04T21:40:59.523 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T21:40:59.533 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-04T21:40:59.550 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-04T21:40:59.645 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-04T21:40:59.700 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-09-04T21:40:59.700 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-09-04T21:40:59.865 INFO:teuthology.orchestra.run.smithi157.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-04T21:40:59.865 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T21:40:59.865 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T21:40:59.865 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T21:40:59.865 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T21:40:59.866 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-04T21:40:59.866 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-04T21:40:59.866 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T21:40:59.907 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-04T21:40:59.909 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2024-09-04T21:40:59.966 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2024-09-04T21:40:59.966 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2024-09-04T21:40:59.966 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 21 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.119.119 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2024-09-04T21:40:59.967 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2024-09-04T21:40:59.968 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-09-04T21:41:00.014 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.ijPR271j6g 2024-09-04T21:41:00.015 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ijPR271j6g 2024-09-04T21:41:00.069 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.ijPR271j6g 2024-09-04T21:41:00.177 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.ijPR271j6g is 10KB 2024-09-04T21:41:00.190 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.ijPR271j6g 2024-09-04T21:41:00.195 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-04T21:41:00.250 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:41:00.250 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-04T21:41:00.251 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-04T21:41:00.318 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-04T21:41:00.318 DEBUG:teuthology.orchestra.run.smithi157:> sudo update-grub 2024-09-04T21:41:00.846 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-04T21:41:00.865 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-04T21:41:00.924 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-04T21:41:01.142 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-04T21:41:01.142 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-04T21:41:01.319 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-04T21:41:01.320 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T21:41:01.360 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-04T21:41:01.362 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 21 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.15.0.119.119 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-04T21:41:01.378 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-04T21:41:01.379 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-04T21:41:01.379 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-04T21:41:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-04T21:41:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-04T21:41:01.379 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-04T21:41:01.425 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.3JLwezbHOR 2024-09-04T21:41:01.425 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3JLwezbHOR 2024-09-04T21:41:01.480 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.3JLwezbHOR 2024-09-04T21:41:01.578 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub' 2024-09-04T21:41:01.580 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-04T21:41:01.582 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-09-04T21:41:01.594 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.3JLwezbHOR is 10KB 2024-09-04T21:41:01.606 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.3JLwezbHOR 2024-09-04T21:41:01.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-04T21:41:01.665 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:41:01.665 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-04T21:41:01.665 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-04T21:41:01.751 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-04T21:41:01.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo update-grub 2024-09-04T21:41:01.786 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-04T21:41:01.792 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-04T21:41:02.015 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-04T21:41:02.021 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-04T21:41:02.062 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-04T21:41:02.067 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-04T21:41:02.225 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-04T21:41:02.225 INFO:teuthology.orchestra.run.smithi157.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-04T21:41:02.225 INFO:teuthology.orchestra.run.smithi157.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-04T21:41:02.238 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-09-04T21:41:02.262 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-09-04T21:41:03.276 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub' 2024-09-04T21:41:03.277 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-04T21:41:03.281 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-09-04T21:41:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-04T21:41:03.522 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-04T21:41:03.766 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-04T21:41:03.771 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-04T21:41:03.811 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-04T21:41:03.816 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-04T21:41:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-04T21:41:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-04T21:41:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-04T21:41:03.959 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-09-04T21:41:03.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-09-04T21:41:32.293 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-04T21:41:32.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:41:32.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:41:34.007 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-04T21:41:34.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:41:34.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:41:50.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-04T21:41:52.573 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-04T21:41:59.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:41:59.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:42:01.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:42:01.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:42:02.845 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-04T21:42:04.641 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-04T21:42:14.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:42:14.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:42:16.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:42:16.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:42:17.917 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-04T21:42:32.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:42:32.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:42:35.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-04T21:42:50.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:42:50.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:42:50.375 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-04T21:42:50.916 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:42:50.916 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-04T21:42:50.916 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-04T21:42:50.962 INFO:teuthology.orchestra.run.smithi112.stdout:5.15.0-119-generic 2024-09-04T21:42:50.962 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-04T21:42:50.962 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-04T21:42:50.962 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-04T21:42:51.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-04T21:42:51.963 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-04T21:42:51.964 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-04T21:42:52.033 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-09-04T21:42:52.068 DEBUG:teuthology.parallel:result is None 2024-09-04T21:43:09.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:43:09.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:43:09.833 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-04T21:43:10.414 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:43:10.414 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-119-generic"... 2024-09-04T21:43:10.415 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-04T21:43:10.462 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-119-generic 2024-09-04T21:43:10.462 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-04T21:43:10.462 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-04T21:43:10.462 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-04T21:43:11.463 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-04T21:43:11.463 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-04T21:43:11.801 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-09-04T21:43:11.840 DEBUG:teuthology.parallel:result is None 2024-09-04T21:43:11.840 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-04T21:43:11.854 INFO:teuthology.task.internal:Creating test directory... 2024-09-04T21:43:11.854 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-04T21:43:11.856 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-04T21:43:11.861 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-04T21:43:11.869 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-04T21:43:11.875 INFO:teuthology.task.internal:Creating archive directory... 2024-09-04T21:43:11.875 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-04T21:43:11.903 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-04T21:43:11.935 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-04T21:43:11.941 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-04T21:43:11.942 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-04T21:43:11.963 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-04T21:43:11.978 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T21:43:11.986 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T21:43:11.988 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T21:43:11.998 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T21:43:11.999 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-04T21:43:12.006 INFO:teuthology.task.internal:Configuring sudo... 2024-09-04T21:43:12.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-04T21:43:12.034 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-04T21:43:12.054 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-04T21:43:12.063 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-04T21:43:12.064 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-04T21:43:12.087 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-04T21:43:12.106 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-04T21:43:12.138 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-04T21:43:12.186 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:43:12.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-04T21:43:12.249 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-04T21:43:12.255 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-04T21:43:12.302 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:43:12.302 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-04T21:43:12.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-09-04T21:43:12.360 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-09-04T21:43:12.432 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-04T21:43:12.440 INFO:teuthology.task.internal:Starting timer... 2024-09-04T21:43:12.440 INFO:teuthology.run_tasks:Running task pcp... 2024-09-04T21:43:12.452 INFO:teuthology.run_tasks:Running task selinux... 2024-09-04T21:43:12.460 DEBUG:teuthology.task.selinux:Excluding smithi112: OS 'ubuntu' does not support SELinux 2024-09-04T21:43:12.460 DEBUG:teuthology.task.selinux:Excluding smithi157: OS 'ubuntu' does not support SELinux 2024-09-04T21:43:12.460 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-04T21:43:12.460 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-04T21:43:12.460 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-04T21:43:12.461 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-04T21:43:12.468 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-04T21:43:12.471 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-04T21:43:12.585 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-04T21:43:12.596 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-04T21:43:12.597 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-04T21:50:38.582 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-09-04T21:50:38.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:50:38.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:50:38.684 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-04T21:50:38.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-04T21:50:38.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:50:38.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T21:50:38.854 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-04T21:50:38.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-04T21:50:38.933 INFO:teuthology.run_tasks:Running task clock... 2024-09-04T21:50:38.945 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-04T21:50:38.945 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-04T21:50:38.946 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-04T21:50:38.948 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-04T21:50:38.948 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-04T21:50:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-04T21:50:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Command line: ntpd -gq 2024-09-04T21:50:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: ---------------------------------------------------- 2024-09-04T21:50:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: ntp-4 is maintained by Network Time Foundation, 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: corporation. Support and training for ntp-4 are 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: available at https://www.nwtime.org/support 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: ---------------------------------------------------- 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: proto: precision = 0.063 usec (-24) 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: basedate set to 2022-02-04 2024-09-04T21:50:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: gps base set to 2022-02-06 (week 2196) 2024-09-04T21:50:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-04T21:50:38.979 INFO:teuthology.orchestra.run.smithi112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-04T21:50:38.979 INFO:teuthology.orchestra.run.smithi112.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-04T21:50:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: restrict ::: KOD does nothing without LIMITED. 2024-09-04T21:50:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listen and drop on 0 v6wildcard [::]:123 2024-09-04T21:50:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-04T21:50:38.981 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listen normally on 2 lo 127.0.0.1:123 2024-09-04T21:50:38.981 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listen normally on 3 enp3s0f1 172.21.15.112:123 2024-09-04T21:50:38.981 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listen normally on 4 lo [::1]:123 2024-09-04T21:50:38.981 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb69%5]:123 2024-09-04T21:50:38.981 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:38 ntpd[18426]: Listening on routing socket on fd #22 for interface updates 2024-09-04T21:50:39.001 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-04T21:50:39.001 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: Command line: ntpd -gq 2024-09-04T21:50:39.002 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: ---------------------------------------------------- 2024-09-04T21:50:39.002 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: ntp-4 is maintained by Network Time Foundation, 2024-09-04T21:50:39.002 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-04T21:50:39.002 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: corporation. Support and training for ntp-4 are 2024-09-04T21:50:39.002 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: available at https://www.nwtime.org/support 2024-09-04T21:50:39.002 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: ---------------------------------------------------- 2024-09-04T21:50:39.003 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: proto: precision = 0.047 usec (-24) 2024-09-04T21:50:39.003 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: basedate set to 2022-02-04 2024-09-04T21:50:39.003 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: gps base set to 2022-02-06 (week 2196) 2024-09-04T21:50:39.003 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-04T21:50:39.004 INFO:teuthology.orchestra.run.smithi157.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-04T21:50:39.004 INFO:teuthology.orchestra.run.smithi157.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:38 ntpd[18455]: restrict ::: KOD does nothing without LIMITED. 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listen and drop on 0 v6wildcard [::]:123 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listen normally on 2 lo 127.0.0.1:123 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listen normally on 3 enp3s0f1 172.21.15.157:123 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listen normally on 4 lo [::1]:123 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea1%5]:123 2024-09-04T21:50:39.005 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:39 ntpd[18455]: Listening on routing socket on fd #22 for interface updates 2024-09-04T21:50:39.977 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:39 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:39.977 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:39 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:40.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:40 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:40.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:40 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:40.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:40 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:40.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:40 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:40.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:40 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:40.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:40 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:42.002 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.002 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:42.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:41 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:42.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:42.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:42.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:42.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:42.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:42.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:42 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:43.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:43 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:43.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:43 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:43.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:43 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:43.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:43 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:43.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:43 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:43.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:43 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:44.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:43 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:44 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:44 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:44 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:44 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:44.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:44 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:44.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:44.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:44 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:45.977 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:45 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:45.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:45 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:45.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:45 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:45.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:45 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:45.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:45 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:45.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:45 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:46.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:46.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:45 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:46 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:46 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:46 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:46.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:46 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:46.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.978 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: 4 Sep 21:50:46 ntpd[18426]: ntpd: time slew -0.001138 s 2024-09-04T21:50:46.980 INFO:teuthology.orchestra.run.smithi112.stdout:ntpd: time slew -0.001138s 2024-09-04T21:50:46.980 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-04T21:50:46.980 INFO:teuthology.orchestra.run.smithi112.stderr: 4 Sep 21:50:46 ntpd[18426]: can't open /var/log/ntpstats/loopstats.20240904: Permission denied 2024-09-04T21:50:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-04T21:50:47.021 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-04T21:50:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:48.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:48.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:48.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:48.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:48.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:48.003 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:47 ntpd[18455]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:48 ntpd[18455]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T21:50:48.004 INFO:teuthology.orchestra.run.smithi157.stdout: 4 Sep 21:50:48 ntpd[18455]: ntpd: time slew -0.000891 s 2024-09-04T21:50:48.005 INFO:teuthology.orchestra.run.smithi157.stdout:ntpd: time slew -0.000891s 2024-09-04T21:50:48.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:48 ntpd[18455]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-04T21:50:48.005 INFO:teuthology.orchestra.run.smithi157.stderr: 4 Sep 21:50:48 ntpd[18455]: can't open /var/log/ntpstats/loopstats.20240904: Permission denied 2024-09-04T21:50:48.048 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2024-09-04T21:50:48.049 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2024-09-04T21:50:48.049 INFO:teuthology.orchestra.run.smithi157.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:48.049 INFO:teuthology.orchestra.run.smithi157.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:48.049 INFO:teuthology.orchestra.run.smithi157.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:48.049 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:50:48.049 INFO:teuthology.run_tasks:Running task install... 2024-09-04T21:50:48.058 DEBUG:teuthology.task.install:project ceph 2024-09-04T21:50:48.058 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab'}} 2024-09-04T21:50:48.058 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab'} 2024-09-04T21:50:48.058 INFO:teuthology.task.install:Using flavor: default 2024-09-04T21:50:48.064 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-04T21:50:48.065 INFO:teuthology.task.install:extra packages: [] 2024-09-04T21:50:48.065 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-key list | grep Ceph 2024-09-04T21:50:48.065 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-key list | grep Ceph 2024-09-04T21:50:48.130 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-04T21:50:48.145 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-04T21:50:48.158 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-04T21:50:48.159 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph.com (release key) 2024-09-04T21:50:48.159 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-04T21:50:48.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab 2024-09-04T21:50:48.169 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-04T21:50:48.170 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph.com (release key) 2024-09-04T21:50:48.170 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-04T21:50:48.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab 2024-09-04T21:50:48.300 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default/ 2024-09-04T21:50:48.301 INFO:teuthology.task.install.deb:Package version is 19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:50:48.305 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default/ 2024-09-04T21:50:48.305 INFO:teuthology.task.install.deb:Package version is 19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:50:48.409 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:50:48.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-04T21:50:48.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-04T21:50:48.425 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:50:48.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-04T21:50:48.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-09-04T21:50:48.611 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-04T21:50:48.645 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy InRelease 2024-09-04T21:50:48.661 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy InRelease 2024-09-04T21:50:48.665 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-04T21:50:48.682 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-04T21:50:48.687 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-04T21:50:48.705 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-04T21:50:48.708 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-04T21:50:48.715 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T21:50:48.726 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,424 B] 2024-09-04T21:50:48.736 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,424 B] 2024-09-04T21:50:48.745 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T21:50:48.745 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-04T21:50:48.766 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-04T21:50:48.795 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-04T21:50:48.810 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-04T21:50:49.152 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-04T21:50:49.165 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-04T21:50:49.243 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-04T21:50:49.258 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-04T21:50:53.398 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 183 kB in 1s (252 kB/s) 2024-09-04T21:50:53.416 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 312 kB in 1s (431 kB/s) 2024-09-04T21:50:54.445 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-04T21:50:54.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4703-g3e4a7f4c-1jammy cephadm=19.3.0-4703-g3e4a7f4c-1jammy ceph-mds=19.3.0-4703-g3e4a7f4c-1jammy ceph-mgr=19.3.0-4703-g3e4a7f4c-1jammy ceph-common=19.3.0-4703-g3e4a7f4c-1jammy ceph-fuse=19.3.0-4703-g3e4a7f4c-1jammy ceph-test=19.3.0-4703-g3e4a7f4c-1jammy ceph-volume=19.3.0-4703-g3e4a7f4c-1jammy radosgw=19.3.0-4703-g3e4a7f4c-1jammy python3-rados=19.3.0-4703-g3e4a7f4c-1jammy python3-rgw=19.3.0-4703-g3e4a7f4c-1jammy python3-cephfs=19.3.0-4703-g3e4a7f4c-1jammy python3-rbd=19.3.0-4703-g3e4a7f4c-1jammy libcephfs2=19.3.0-4703-g3e4a7f4c-1jammy libcephfs-dev=19.3.0-4703-g3e4a7f4c-1jammy librados2=19.3.0-4703-g3e4a7f4c-1jammy librbd1=19.3.0-4703-g3e4a7f4c-1jammy rbd-fuse=19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:50:54.477 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-04T21:50:54.494 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4703-g3e4a7f4c-1jammy cephadm=19.3.0-4703-g3e4a7f4c-1jammy ceph-mds=19.3.0-4703-g3e4a7f4c-1jammy ceph-mgr=19.3.0-4703-g3e4a7f4c-1jammy ceph-common=19.3.0-4703-g3e4a7f4c-1jammy ceph-fuse=19.3.0-4703-g3e4a7f4c-1jammy ceph-test=19.3.0-4703-g3e4a7f4c-1jammy ceph-volume=19.3.0-4703-g3e4a7f4c-1jammy radosgw=19.3.0-4703-g3e4a7f4c-1jammy python3-rados=19.3.0-4703-g3e4a7f4c-1jammy python3-rgw=19.3.0-4703-g3e4a7f4c-1jammy python3-cephfs=19.3.0-4703-g3e4a7f4c-1jammy python3-rbd=19.3.0-4703-g3e4a7f4c-1jammy libcephfs2=19.3.0-4703-g3e4a7f4c-1jammy libcephfs-dev=19.3.0-4703-g3e4a7f4c-1jammy librados2=19.3.0-4703-g3e4a7f4c-1jammy librbd1=19.3.0-4703-g3e4a7f4c-1jammy rbd-fuse=19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:50:54.523 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-04T21:50:54.547 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-04T21:50:54.731 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-04T21:50:54.731 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-04T21:50:54.733 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-09-04T21:50:54.734 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-09-04T21:50:54.916 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T21:50:54.916 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T21:50:54.916 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T21:50:54.917 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T21:50:54.917 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-04T21:50:54.917 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-04T21:50:54.917 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-04T21:50:54.919 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-04T21:50:54.920 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-04T21:50:54.921 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet zip 2024-09-04T21:50:54.922 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-09-04T21:50:54.922 INFO:teuthology.orchestra.run.smithi157.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-04T21:50:54.922 INFO:teuthology.orchestra.run.smithi157.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-04T21:50:54.922 INFO:teuthology.orchestra.run.smithi157.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-04T21:50:54.922 INFO:teuthology.orchestra.run.smithi157.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-04T21:50:54.922 INFO:teuthology.orchestra.run.smithi157.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-04T21:50:54.923 INFO:teuthology.orchestra.run.smithi157.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-04T21:50:54.923 INFO:teuthology.orchestra.run.smithi157.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-04T21:50:54.923 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2024-09-04T21:50:54.923 INFO:teuthology.orchestra.run.smithi157.stdout: btrfs-tools 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T21:50:54.925 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-04T21:50:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-04T21:50:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet zip 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-04T21:50:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: btrfs-tools 2024-09-04T21:50:55.022 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-09-04T21:50:55.023 INFO:teuthology.orchestra.run.smithi157.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-04T21:50:55.023 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-04T21:50:55.023 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-04T21:50:55.023 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-04T21:50:55.023 INFO:teuthology.orchestra.run.smithi157.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-04T21:50:55.023 INFO:teuthology.orchestra.run.smithi157.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-04T21:50:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-04T21:50:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-04T21:50:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-04T21:50:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-04T21:50:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-04T21:50:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-04T21:50:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-04T21:50:55.030 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-04T21:50:55.030 INFO:teuthology.orchestra.run.smithi112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-04T21:50:55.030 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-04T21:50:55.030 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-04T21:50:55.032 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-04T21:50:55.033 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-04T21:50:55.034 INFO:teuthology.orchestra.run.smithi112.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-04T21:50:55.084 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-04T21:50:55.084 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 251 MB of archives. 2024-09-04T21:50:55.084 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-04T21:50:55.084 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-04T21:50:55.166 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4703-g3e4a7f4c-1jammy [3,631 kB] 2024-09-04T21:50:55.172 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-04T21:50:55.173 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-04T21:50:55.173 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 251 MB of archives. 2024-09-04T21:50:55.174 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-04T21:50:55.174 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4703-g3e4a7f4c-1jammy [3,631 kB] 2024-09-04T21:50:55.174 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-04T21:50:55.191 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-04T21:50:55.219 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-04T21:50:55.251 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-04T21:50:55.256 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-04T21:50:55.267 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-04T21:50:55.271 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-04T21:50:55.273 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-04T21:50:55.273 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-04T21:50:55.274 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-04T21:50:55.282 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-04T21:50:55.284 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-04T21:50:55.286 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-04T21:50:55.289 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-04T21:50:55.289 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-04T21:50:55.306 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-04T21:50:55.308 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-04T21:50:55.310 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-04T21:50:55.310 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-04T21:50:55.311 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-04T21:50:55.312 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-04T21:50:55.312 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-04T21:50:55.313 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-04T21:50:55.323 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-04T21:50:55.324 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-04T21:50:55.340 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-04T21:50:55.341 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-04T21:50:55.343 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-04T21:50:55.343 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-04T21:50:55.344 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-04T21:50:55.344 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-04T21:50:55.345 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-04T21:50:55.345 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-04T21:50:55.357 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-04T21:50:55.358 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-04T21:50:55.374 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-04T21:50:55.375 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-04T21:50:55.380 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-04T21:50:55.380 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-04T21:50:55.381 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-04T21:50:55.381 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-04T21:50:55.382 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-04T21:50:55.384 INFO:teuthology.orchestra.run.smithi157.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-04T21:50:55.391 INFO:teuthology.orchestra.run.smithi157.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-04T21:50:55.392 INFO:teuthology.orchestra.run.smithi157.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-04T21:50:55.402 INFO:teuthology.orchestra.run.smithi157.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-04T21:50:55.403 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4703-g3e4a7f4c-1jammy [3,335 kB] 2024-09-04T21:50:55.405 INFO:teuthology.orchestra.run.smithi157.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-04T21:50:55.405 INFO:teuthology.orchestra.run.smithi157.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-04T21:50:55.406 INFO:teuthology.orchestra.run.smithi157.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-04T21:50:55.408 INFO:teuthology.orchestra.run.smithi157.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-04T21:50:55.419 INFO:teuthology.orchestra.run.smithi157.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4703-g3e4a7f4c-1jammy [3,335 kB] 2024-09-04T21:50:55.459 INFO:teuthology.orchestra.run.smithi157.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-04T21:50:55.460 INFO:teuthology.orchestra.run.smithi157.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-04T21:50:55.461 INFO:teuthology.orchestra.run.smithi157.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-04T21:50:55.461 INFO:teuthology.orchestra.run.smithi157.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-04T21:50:55.463 INFO:teuthology.orchestra.run.smithi157.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-04T21:50:55.479 INFO:teuthology.orchestra.run.smithi157.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-04T21:50:55.481 INFO:teuthology.orchestra.run.smithi157.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-04T21:50:55.481 INFO:teuthology.orchestra.run.smithi157.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-04T21:50:55.496 INFO:teuthology.orchestra.run.smithi157.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-04T21:50:55.496 INFO:teuthology.orchestra.run.smithi157.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-04T21:50:55.496 INFO:teuthology.orchestra.run.smithi157.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-04T21:50:55.497 INFO:teuthology.orchestra.run.smithi157.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-04T21:50:55.497 INFO:teuthology.orchestra.run.smithi157.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-04T21:50:55.498 INFO:teuthology.orchestra.run.smithi157.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-04T21:50:55.503 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4703-g3e4a7f4c-1jammy [722 kB] 2024-09-04T21:50:55.512 INFO:teuthology.orchestra.run.smithi157.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-04T21:50:55.513 INFO:teuthology.orchestra.run.smithi157.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-04T21:50:55.516 INFO:teuthology.orchestra.run.smithi157.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-04T21:50:55.517 INFO:teuthology.orchestra.run.smithi157.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4703-g3e4a7f4c-1jammy [722 kB] 2024-09-04T21:50:55.526 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4703-g3e4a7f4c-1jammy [356 kB] 2024-09-04T21:50:55.530 INFO:teuthology.orchestra.run.smithi157.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-04T21:50:55.532 INFO:teuthology.orchestra.run.smithi157.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-04T21:50:55.532 INFO:teuthology.orchestra.run.smithi157.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-04T21:50:55.533 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4703-g3e4a7f4c-1jammy [32.8 kB] 2024-09-04T21:50:55.535 INFO:teuthology.orchestra.run.smithi157.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-04T21:50:55.537 INFO:teuthology.orchestra.run.smithi157.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-04T21:50:55.538 INFO:teuthology.orchestra.run.smithi157.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-04T21:50:55.538 INFO:teuthology.orchestra.run.smithi157.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-04T21:50:55.541 INFO:teuthology.orchestra.run.smithi157.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4703-g3e4a7f4c-1jammy [356 kB] 2024-09-04T21:50:55.542 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4703-g3e4a7f4c-1jammy [180 kB] 2024-09-04T21:50:55.547 INFO:teuthology.orchestra.run.smithi157.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-04T21:50:55.548 INFO:teuthology.orchestra.run.smithi157.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-04T21:50:55.550 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4703-g3e4a7f4c-1jammy [71.2 kB] 2024-09-04T21:50:55.550 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4703-g3e4a7f4c-1jammy [338 kB] 2024-09-04T21:50:55.551 INFO:teuthology.orchestra.run.smithi157.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-04T21:50:55.557 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4703-g3e4a7f4c-1jammy [7,979 kB] 2024-09-04T21:50:55.558 INFO:teuthology.orchestra.run.smithi157.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4703-g3e4a7f4c-1jammy [32.8 kB] 2024-09-04T21:50:55.559 INFO:teuthology.orchestra.run.smithi157.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4703-g3e4a7f4c-1jammy [180 kB] 2024-09-04T21:50:55.563 INFO:teuthology.orchestra.run.smithi157.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4703-g3e4a7f4c-1jammy [71.2 kB] 2024-09-04T21:50:55.564 INFO:teuthology.orchestra.run.smithi157.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-04T21:50:55.565 INFO:teuthology.orchestra.run.smithi157.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-04T21:50:55.565 INFO:teuthology.orchestra.run.smithi157.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4703-g3e4a7f4c-1jammy [338 kB] 2024-09-04T21:50:55.565 INFO:teuthology.orchestra.run.smithi157.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-04T21:50:55.566 INFO:teuthology.orchestra.run.smithi157.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-04T21:50:55.567 INFO:teuthology.orchestra.run.smithi157.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-04T21:50:55.572 INFO:teuthology.orchestra.run.smithi157.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4703-g3e4a7f4c-1jammy [7,979 kB] 2024-09-04T21:50:55.573 INFO:teuthology.orchestra.run.smithi157.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-04T21:50:55.573 INFO:teuthology.orchestra.run.smithi157.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-04T21:50:55.581 INFO:teuthology.orchestra.run.smithi157.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-04T21:50:55.583 INFO:teuthology.orchestra.run.smithi157.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-04T21:50:55.583 INFO:teuthology.orchestra.run.smithi157.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-04T21:50:55.640 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-04T21:50:55.652 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-04T21:50:55.733 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-04T21:50:55.838 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4703-g3e4a7f4c-1jammy [112 kB] 2024-09-04T21:50:55.841 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4703-g3e4a7f4c-1jammy [473 kB] 2024-09-04T21:50:55.846 INFO:teuthology.orchestra.run.smithi157.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4703-g3e4a7f4c-1jammy [112 kB] 2024-09-04T21:50:55.849 INFO:teuthology.orchestra.run.smithi157.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4703-g3e4a7f4c-1jammy [473 kB] 2024-09-04T21:50:55.851 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4703-g3e4a7f4c-1jammy [28.2 MB] 2024-09-04T21:50:55.859 INFO:teuthology.orchestra.run.smithi157.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4703-g3e4a7f4c-1jammy [28.2 MB] 2024-09-04T21:50:55.962 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-04T21:50:55.972 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-04T21:50:56.002 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-04T21:50:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-04T21:50:56.013 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-04T21:50:56.013 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-04T21:50:56.015 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-04T21:50:56.032 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-04T21:50:56.037 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-04T21:50:56.043 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-04T21:50:56.122 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-04T21:50:56.123 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-04T21:50:56.127 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-04T21:50:56.131 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-04T21:50:56.134 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-04T21:50:56.135 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-04T21:50:56.136 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-04T21:50:56.138 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-04T21:50:56.138 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-04T21:50:56.206 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-04T21:50:56.221 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-04T21:50:56.221 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-04T21:50:56.222 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-04T21:50:56.223 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-04T21:50:56.228 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-04T21:50:56.229 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-04T21:50:56.288 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-04T21:50:56.288 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-04T21:50:56.290 INFO:teuthology.orchestra.run.smithi112.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-04T21:50:56.369 INFO:teuthology.orchestra.run.smithi112.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-04T21:50:56.369 INFO:teuthology.orchestra.run.smithi112.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-04T21:50:56.371 INFO:teuthology.orchestra.run.smithi112.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-04T21:50:56.373 INFO:teuthology.orchestra.run.smithi112.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-04T21:50:56.373 INFO:teuthology.orchestra.run.smithi112.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-04T21:50:56.384 INFO:teuthology.orchestra.run.smithi112.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-04T21:50:56.449 INFO:teuthology.orchestra.run.smithi112.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-04T21:50:56.450 INFO:teuthology.orchestra.run.smithi112.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-04T21:50:56.451 INFO:teuthology.orchestra.run.smithi112.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-04T21:50:56.454 INFO:teuthology.orchestra.run.smithi112.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-04T21:50:56.531 INFO:teuthology.orchestra.run.smithi112.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-04T21:50:56.532 INFO:teuthology.orchestra.run.smithi112.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-04T21:50:56.535 INFO:teuthology.orchestra.run.smithi112.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-04T21:50:56.536 INFO:teuthology.orchestra.run.smithi112.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-04T21:50:56.541 INFO:teuthology.orchestra.run.smithi112.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-04T21:50:56.542 INFO:teuthology.orchestra.run.smithi112.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-04T21:50:56.612 INFO:teuthology.orchestra.run.smithi112.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-04T21:50:56.616 INFO:teuthology.orchestra.run.smithi112.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-04T21:50:56.779 INFO:teuthology.orchestra.run.smithi112.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-04T21:50:56.781 INFO:teuthology.orchestra.run.smithi112.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-04T21:50:56.784 INFO:teuthology.orchestra.run.smithi112.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-04T21:50:56.784 INFO:teuthology.orchestra.run.smithi112.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-04T21:50:56.787 INFO:teuthology.orchestra.run.smithi112.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-04T21:50:56.850 INFO:teuthology.orchestra.run.smithi112.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-04T21:50:56.917 INFO:teuthology.orchestra.run.smithi157.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4703-g3e4a7f4c-1jammy [5,203 kB] 2024-09-04T21:50:56.945 INFO:teuthology.orchestra.run.smithi112.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-04T21:50:56.945 INFO:teuthology.orchestra.run.smithi112.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-04T21:50:56.946 INFO:teuthology.orchestra.run.smithi112.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4703-g3e4a7f4c-1jammy [5,203 kB] 2024-09-04T21:50:56.974 INFO:teuthology.orchestra.run.smithi112.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-04T21:50:56.975 INFO:teuthology.orchestra.run.smithi112.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-04T21:50:56.976 INFO:teuthology.orchestra.run.smithi112.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-04T21:50:56.977 INFO:teuthology.orchestra.run.smithi112.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-04T21:50:56.977 INFO:teuthology.orchestra.run.smithi112.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-04T21:50:56.978 INFO:teuthology.orchestra.run.smithi112.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-04T21:50:56.985 INFO:teuthology.orchestra.run.smithi157.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4703-g3e4a7f4c-1jammy [251 kB] 2024-09-04T21:50:56.988 INFO:teuthology.orchestra.run.smithi157.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4703-g3e4a7f4c-1jammy [124 kB] 2024-09-04T21:50:56.990 INFO:teuthology.orchestra.run.smithi157.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4703-g3e4a7f4c-1jammy [1,561 kB] 2024-09-04T21:50:57.022 INFO:teuthology.orchestra.run.smithi157.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4703-g3e4a7f4c-1jammy [6,309 kB] 2024-09-04T21:50:57.028 INFO:teuthology.orchestra.run.smithi112.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-04T21:50:57.028 INFO:teuthology.orchestra.run.smithi112.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4703-g3e4a7f4c-1jammy [251 kB] 2024-09-04T21:50:57.029 INFO:teuthology.orchestra.run.smithi112.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-04T21:50:57.030 INFO:teuthology.orchestra.run.smithi112.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4703-g3e4a7f4c-1jammy [124 kB] 2024-09-04T21:50:57.032 INFO:teuthology.orchestra.run.smithi112.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4703-g3e4a7f4c-1jammy [1,561 kB] 2024-09-04T21:50:57.085 INFO:teuthology.orchestra.run.smithi112.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4703-g3e4a7f4c-1jammy [6,309 kB] 2024-09-04T21:50:57.109 INFO:teuthology.orchestra.run.smithi112.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-04T21:50:57.113 INFO:teuthology.orchestra.run.smithi112.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-04T21:50:57.117 INFO:teuthology.orchestra.run.smithi112.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-04T21:50:57.118 INFO:teuthology.orchestra.run.smithi112.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-04T21:50:57.127 INFO:teuthology.orchestra.run.smithi112.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-04T21:50:57.133 INFO:teuthology.orchestra.run.smithi112.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-04T21:50:57.191 INFO:teuthology.orchestra.run.smithi112.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-04T21:50:57.192 INFO:teuthology.orchestra.run.smithi112.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-04T21:50:57.203 INFO:teuthology.orchestra.run.smithi112.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-04T21:50:57.204 INFO:teuthology.orchestra.run.smithi112.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-04T21:50:57.216 INFO:teuthology.orchestra.run.smithi157.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4703-g3e4a7f4c-1jammy [23.1 MB] 2024-09-04T21:50:57.272 INFO:teuthology.orchestra.run.smithi112.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-04T21:50:57.272 INFO:teuthology.orchestra.run.smithi112.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-04T21:50:57.273 INFO:teuthology.orchestra.run.smithi112.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-04T21:50:57.274 INFO:teuthology.orchestra.run.smithi112.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-04T21:50:57.278 INFO:teuthology.orchestra.run.smithi112.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-04T21:50:57.280 INFO:teuthology.orchestra.run.smithi112.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-04T21:50:57.281 INFO:teuthology.orchestra.run.smithi112.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4703-g3e4a7f4c-1jammy [23.1 MB] 2024-09-04T21:50:57.353 INFO:teuthology.orchestra.run.smithi112.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-04T21:50:57.353 INFO:teuthology.orchestra.run.smithi112.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-04T21:50:57.354 INFO:teuthology.orchestra.run.smithi112.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-04T21:50:57.359 INFO:teuthology.orchestra.run.smithi112.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-04T21:50:57.434 INFO:teuthology.orchestra.run.smithi112.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-04T21:50:58.068 INFO:teuthology.orchestra.run.smithi157.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4703-g3e4a7f4c-1jammy [14.0 kB] 2024-09-04T21:50:58.069 INFO:teuthology.orchestra.run.smithi157.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4703-g3e4a7f4c-1jammy [880 kB] 2024-09-04T21:50:58.087 INFO:teuthology.orchestra.run.smithi157.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4703-g3e4a7f4c-1jammy [2,433 kB] 2024-09-04T21:50:58.130 INFO:teuthology.orchestra.run.smithi157.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4703-g3e4a7f4c-1jammy [812 kB] 2024-09-04T21:50:58.140 INFO:teuthology.orchestra.run.smithi157.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4703-g3e4a7f4c-1jammy [160 kB] 2024-09-04T21:50:58.141 INFO:teuthology.orchestra.run.smithi157.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4703-g3e4a7f4c-1jammy [61.4 MB] 2024-09-04T21:50:58.144 INFO:teuthology.orchestra.run.smithi112.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4703-g3e4a7f4c-1jammy [14.0 kB] 2024-09-04T21:50:58.144 INFO:teuthology.orchestra.run.smithi112.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4703-g3e4a7f4c-1jammy [880 kB] 2024-09-04T21:50:58.162 INFO:teuthology.orchestra.run.smithi112.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4703-g3e4a7f4c-1jammy [2,433 kB] 2024-09-04T21:50:58.202 INFO:teuthology.orchestra.run.smithi112.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4703-g3e4a7f4c-1jammy [812 kB] 2024-09-04T21:50:58.212 INFO:teuthology.orchestra.run.smithi112.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4703-g3e4a7f4c-1jammy [160 kB] 2024-09-04T21:50:58.214 INFO:teuthology.orchestra.run.smithi112.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4703-g3e4a7f4c-1jammy [61.4 MB] 2024-09-04T21:51:00.415 INFO:teuthology.orchestra.run.smithi157.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4703-g3e4a7f4c-1jammy [8,625 kB] 2024-09-04T21:51:00.527 INFO:teuthology.orchestra.run.smithi157.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4703-g3e4a7f4c-1jammy [14.2 kB] 2024-09-04T21:51:00.527 INFO:teuthology.orchestra.run.smithi157.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4703-g3e4a7f4c-1jammy [55.7 MB] 2024-09-04T21:51:00.530 INFO:teuthology.orchestra.run.smithi112.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4703-g3e4a7f4c-1jammy [8,625 kB] 2024-09-04T21:51:00.728 INFO:teuthology.orchestra.run.smithi112.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4703-g3e4a7f4c-1jammy [14.2 kB] 2024-09-04T21:51:00.729 INFO:teuthology.orchestra.run.smithi112.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4703-g3e4a7f4c-1jammy [55.7 MB] 2024-09-04T21:51:02.519 INFO:teuthology.orchestra.run.smithi157.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4703-g3e4a7f4c-1jammy [130 kB] 2024-09-04T21:51:02.522 INFO:teuthology.orchestra.run.smithi157.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4703-g3e4a7f4c-1jammy [38.9 kB] 2024-09-04T21:51:02.587 INFO:teuthology.orchestra.run.smithi157.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4703-g3e4a7f4c-1jammy [15.9 MB] 2024-09-04T21:51:02.766 INFO:teuthology.orchestra.run.smithi112.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4703-g3e4a7f4c-1jammy [130 kB] 2024-09-04T21:51:02.769 INFO:teuthology.orchestra.run.smithi112.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4703-g3e4a7f4c-1jammy [38.9 kB] 2024-09-04T21:51:02.770 INFO:teuthology.orchestra.run.smithi112.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4703-g3e4a7f4c-1jammy [15.9 MB] 2024-09-04T21:51:02.899 INFO:teuthology.orchestra.run.smithi157.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4703-g3e4a7f4c-1jammy [91.0 kB] 2024-09-04T21:51:03.088 INFO:teuthology.orchestra.run.smithi112.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4703-g3e4a7f4c-1jammy [91.0 kB] 2024-09-04T21:51:03.467 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 251 MB in 8s (31.9 MB/s) 2024-09-04T21:51:03.634 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-04T21:51:03.658 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 251 MB in 8s (31.2 MB/s) 2024-09-04T21:51:03.734 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-04T21:51:03.739 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-04T21:51:03.815 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-04T21:51:03.830 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-04T21:51:03.931 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-04T21:51:03.936 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-04T21:51:04.020 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-04T21:51:04.152 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-04T21:51:04.178 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-04T21:51:04.219 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-04T21:51:04.373 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-04T21:51:04.400 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-04T21:51:04.440 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-04T21:51:04.547 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-04T21:51:04.573 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-04T21:51:04.613 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-04T21:51:04.743 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-04T21:51:04.769 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-04T21:51:04.801 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-04T21:51:05.042 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-04T21:51:05.068 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T21:51:05.108 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:51:05.263 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-04T21:51:05.290 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T21:51:05.338 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:51:05.512 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-04T21:51:05.538 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T21:51:05.578 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:51:05.749 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-04T21:51:05.775 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T21:51:05.816 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:51:05.923 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-04T21:51:05.949 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T21:51:05.989 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:51:06.127 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-04T21:51:06.154 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T21:51:06.185 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:51:06.385 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-04T21:51:06.411 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-04T21:51:06.443 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-04T21:51:06.531 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-04T21:51:06.556 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-04T21:51:06.605 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-04T21:51:06.746 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librados2. 2024-09-04T21:51:06.772 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../007-librados2_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:06.812 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librados2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:06.933 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librados2. 2024-09-04T21:51:06.959 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../007-librados2_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:07.000 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librados2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:07.299 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libnbd0. 2024-09-04T21:51:07.325 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-04T21:51:07.365 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-04T21:51:07.660 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librbd1. 2024-09-04T21:51:07.687 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../009-librbd1_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:07.726 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librbd1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:07.987 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libnbd0. 2024-09-04T21:51:08.013 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-04T21:51:08.053 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-04T21:51:08.138 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs2. 2024-09-04T21:51:08.164 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:08.204 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:08.372 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librbd1. 2024-09-04T21:51:08.399 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../009-librbd1_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:08.439 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librbd1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:08.657 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rados. 2024-09-04T21:51:08.672 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:08.715 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rados (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:08.892 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs2. 2024-09-04T21:51:08.918 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:08.958 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:09.026 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-04T21:51:09.053 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:09.093 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-argparse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:09.245 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rados. 2024-09-04T21:51:09.271 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:09.311 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rados (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:09.370 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cephfs. 2024-09-04T21:51:09.397 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:09.437 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cephfs (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:09.632 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-04T21:51:09.657 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:09.697 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-argparse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:09.731 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-04T21:51:09.758 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:09.798 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:09.966 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cephfs. 2024-09-04T21:51:09.981 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:10.016 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cephfs (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:10.092 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-04T21:51:10.119 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-04T21:51:10.159 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-04T21:51:10.311 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-04T21:51:10.337 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:10.377 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:10.487 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-prettytable. 2024-09-04T21:51:10.514 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-04T21:51:10.554 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-04T21:51:10.705 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-04T21:51:10.732 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-04T21:51:10.772 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-04T21:51:10.823 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rbd. 2024-09-04T21:51:10.838 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:10.881 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rbd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:11.092 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-prettytable. 2024-09-04T21:51:11.118 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-04T21:51:11.158 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-04T21:51:11.201 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-04T21:51:11.227 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-04T21:51:11.269 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-04T21:51:11.394 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rbd. 2024-09-04T21:51:11.409 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:11.444 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rbd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:11.547 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-04T21:51:11.563 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-04T21:51:11.605 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-04T21:51:11.747 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-04T21:51:11.762 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-04T21:51:11.805 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-04T21:51:11.891 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-04T21:51:11.907 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-04T21:51:11.950 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-04T21:51:12.066 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-04T21:51:12.081 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-04T21:51:12.149 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-04T21:51:12.261 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua5.1. 2024-09-04T21:51:12.288 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-04T21:51:12.328 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-04T21:51:12.469 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-04T21:51:12.484 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-04T21:51:12.527 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-04T21:51:12.673 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-any. 2024-09-04T21:51:12.688 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-04T21:51:12.731 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-04T21:51:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua5.1. 2024-09-04T21:51:12.899 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-04T21:51:12.939 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-04T21:51:13.000 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package zip. 2024-09-04T21:51:13.027 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-04T21:51:13.067 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking zip (3.0-12build2) ... 2024-09-04T21:51:13.275 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-any. 2024-09-04T21:51:13.301 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-04T21:51:13.342 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-04T21:51:13.395 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package unzip. 2024-09-04T21:51:13.422 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-04T21:51:13.462 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-04T21:51:13.594 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package zip. 2024-09-04T21:51:13.610 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-04T21:51:13.653 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking zip (3.0-12build2) ... 2024-09-04T21:51:13.964 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package unzip. 2024-09-04T21:51:13.974 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package luarocks. 2024-09-04T21:51:13.990 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-04T21:51:14.000 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-04T21:51:14.031 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-04T21:51:14.040 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-04T21:51:14.368 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package luarocks. 2024-09-04T21:51:14.395 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-04T21:51:14.415 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librgw2. 2024-09-04T21:51:14.441 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../026-librgw2_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:14.443 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-04T21:51:14.481 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librgw2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:14.826 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librgw2. 2024-09-04T21:51:14.853 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../026-librgw2_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:14.892 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librgw2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:15.101 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rgw. 2024-09-04T21:51:15.127 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:15.167 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:15.520 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-04T21:51:15.528 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rgw. 2024-09-04T21:51:15.544 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:15.547 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-04T21:51:15.579 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-04T21:51:15.586 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:15.890 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libradosstriper1. 2024-09-04T21:51:15.897 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-04T21:51:15.917 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:15.923 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-04T21:51:15.956 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libradosstriper1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:15.964 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-04T21:51:16.242 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libradosstriper1. 2024-09-04T21:51:16.268 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:16.308 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libradosstriper1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:16.385 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-common. 2024-09-04T21:51:16.411 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:16.451 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:16.762 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-common. 2024-09-04T21:51:16.788 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:16.829 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:17.823 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-base. 2024-09-04T21:51:17.850 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:17.890 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-base (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:18.359 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-base. 2024-09-04T21:51:18.385 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:18.426 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-base (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:18.461 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-04T21:51:18.487 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-04T21:51:18.527 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-04T21:51:19.013 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-04T21:51:19.039 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cheroot. 2024-09-04T21:51:19.039 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-04T21:51:19.066 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-04T21:51:19.080 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-04T21:51:19.105 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-04T21:51:19.425 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-04T21:51:19.432 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cheroot. 2024-09-04T21:51:19.452 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-04T21:51:19.459 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-04T21:51:19.491 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-04T21:51:19.507 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-04T21:51:19.752 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-04T21:51:19.779 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-04T21:51:19.810 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-04T21:51:19.911 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-04T21:51:19.937 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-04T21:51:19.986 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-04T21:51:20.122 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-04T21:51:20.149 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-04T21:51:20.188 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-04T21:51:20.355 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-04T21:51:20.382 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-04T21:51:20.430 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-04T21:51:20.491 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempora. 2024-09-04T21:51:20.518 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-04T21:51:20.558 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-04T21:51:20.733 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-04T21:51:20.759 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-04T21:51:20.791 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-04T21:51:20.836 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-portend. 2024-09-04T21:51:20.863 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-04T21:51:20.902 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-04T21:51:21.078 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempora. 2024-09-04T21:51:21.104 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-04T21:51:21.144 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-04T21:51:21.189 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-04T21:51:21.215 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-04T21:51:21.255 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-04T21:51:21.464 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-portend. 2024-09-04T21:51:21.490 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-04T21:51:21.530 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-04T21:51:21.558 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-04T21:51:21.585 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-04T21:51:21.642 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-04T21:51:21.808 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-04T21:51:21.834 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-04T21:51:21.874 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-04T21:51:21.995 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-natsort. 2024-09-04T21:51:22.021 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-04T21:51:22.061 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-04T21:51:22.144 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-04T21:51:22.170 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-04T21:51:22.211 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-04T21:51:22.340 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-logutils. 2024-09-04T21:51:22.366 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-04T21:51:22.406 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-04T21:51:22.531 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-natsort. 2024-09-04T21:51:22.557 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-04T21:51:22.589 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-04T21:51:22.709 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-04T21:51:22.736 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-04T21:51:22.775 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-04T21:51:23.078 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-mako. 2024-09-04T21:51:23.105 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-04T21:51:23.143 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-logutils. 2024-09-04T21:51:23.153 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-04T21:51:23.169 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-04T21:51:23.242 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-04T21:51:23.482 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-04T21:51:23.509 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-04T21:51:23.548 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-04T21:51:23.612 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-04T21:51:23.638 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-04T21:51:23.687 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-04T21:51:23.784 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-04T21:51:23.811 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-04T21:51:23.842 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-04T21:51:24.056 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-mako. 2024-09-04T21:51:24.083 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-04T21:51:24.131 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-04T21:51:24.162 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webob. 2024-09-04T21:51:24.189 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-04T21:51:24.237 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-04T21:51:24.518 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-04T21:51:24.544 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-04T21:51:24.584 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-04T21:51:24.590 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-waitress. 2024-09-04T21:51:24.617 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-04T21:51:24.660 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-04T21:51:24.904 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-04T21:51:24.930 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-04T21:51:24.979 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-04T21:51:24.993 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempita. 2024-09-04T21:51:25.019 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-04T21:51:25.059 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-04T21:51:25.240 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webob. 2024-09-04T21:51:25.266 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-04T21:51:25.298 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-04T21:51:25.337 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-paste. 2024-09-04T21:51:25.364 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-04T21:51:25.396 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-04T21:51:25.601 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-waitress. 2024-09-04T21:51:25.628 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-04T21:51:25.670 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-04T21:51:25.724 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-04T21:51:25.739 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-04T21:51:25.782 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-04T21:51:25.980 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempita. 2024-09-04T21:51:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-04T21:51:26.026 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-04T21:51:26.046 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-04T21:51:26.053 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-04T21:51:26.085 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-04T21:51:26.338 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webtest. 2024-09-04T21:51:26.365 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-04T21:51:26.404 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-04T21:51:26.407 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-paste. 2024-09-04T21:51:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-04T21:51:26.482 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-04T21:51:26.715 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pecan. 2024-09-04T21:51:26.742 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-04T21:51:26.782 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-04T21:51:26.827 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-04T21:51:26.842 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-04T21:51:26.885 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-04T21:51:27.137 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-certifi. 2024-09-04T21:51:27.164 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-04T21:51:27.188 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-04T21:51:27.203 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-04T21:51:27.215 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-04T21:51:27.255 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-04T21:51:27.531 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-idna. 2024-09-04T21:51:27.558 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webtest. 2024-09-04T21:51:27.558 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-04T21:51:27.584 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-04T21:51:27.598 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-04T21:51:27.624 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-04T21:51:27.884 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-urllib3. 2024-09-04T21:51:27.911 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-04T21:51:27.918 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pecan. 2024-09-04T21:51:27.934 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-04T21:51:27.942 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-04T21:51:27.968 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-04T21:51:28.212 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests. 2024-09-04T21:51:28.240 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-04T21:51:28.270 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-04T21:51:28.665 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-04T21:51:28.692 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-04T21:51:28.740 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-04T21:51:29.026 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-04T21:51:29.053 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:29.216 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-certifi. 2024-09-04T21:51:29.242 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-04T21:51:29.273 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-04T21:51:29.522 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:29.585 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-idna. 2024-09-04T21:51:29.611 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-04T21:51:29.651 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-04T21:51:30.030 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-urllib3. 2024-09-04T21:51:30.056 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-04T21:51:30.096 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-04T21:51:30.423 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-04T21:51:30.424 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests. 2024-09-04T21:51:30.450 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:30.450 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-04T21:51:30.489 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:30.490 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-04T21:51:30.802 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-04T21:51:30.809 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-04T21:51:30.828 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-04T21:51:30.835 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-04T21:51:30.869 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-04T21:51:30.900 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-04T21:51:31.171 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-04T21:51:31.178 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-openssl. 2024-09-04T21:51:31.187 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:31.205 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-04T21:51:31.230 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:31.245 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-04T21:51:31.548 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr. 2024-09-04T21:51:31.564 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:31.598 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:31.601 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-04T21:51:31.628 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:31.668 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:31.943 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mon. 2024-09-04T21:51:31.970 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:31.987 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-04T21:51:32.009 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mon (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:32.014 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-04T21:51:32.054 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-04T21:51:32.340 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-openssl. 2024-09-04T21:51:32.367 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-04T21:51:32.407 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-04T21:51:32.680 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-osd. 2024-09-04T21:51:32.707 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:32.726 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr. 2024-09-04T21:51:32.746 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-osd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:32.753 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:32.793 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:33.221 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mon. 2024-09-04T21:51:33.248 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:33.288 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mon (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:33.782 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph. 2024-09-04T21:51:33.798 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../067-ceph_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:33.832 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:33.883 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-osd. 2024-09-04T21:51:33.910 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:33.966 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-osd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:34.110 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-fuse. 2024-09-04T21:51:34.137 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:34.168 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:34.555 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mds. 2024-09-04T21:51:34.582 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:34.624 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mds (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:34.986 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph. 2024-09-04T21:51:35.013 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../067-ceph_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:35.052 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:35.058 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephadm. 2024-09-04T21:51:35.085 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../070-cephadm_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:35.124 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:35.380 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-fuse. 2024-09-04T21:51:35.402 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-04T21:51:35.407 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:35.417 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-04T21:51:35.447 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:35.451 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-04T21:51:35.780 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-04T21:51:35.792 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mds. 2024-09-04T21:51:35.795 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-04T21:51:35.818 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:35.838 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-04T21:51:35.849 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mds (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:36.228 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package cephadm. 2024-09-04T21:51:36.254 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../070-cephadm_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:36.294 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-04T21:51:36.641 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-04T21:51:36.680 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-04T21:51:36.851 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-babel. 2024-09-04T21:51:36.878 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-04T21:51:36.917 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-04T21:51:37.000 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-04T21:51:37.026 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-04T21:51:37.066 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-04T21:51:37.246 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jinja2. 2024-09-04T21:51:37.272 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-04T21:51:37.312 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-04T21:51:37.615 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-04T21:51:37.630 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:37.665 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:37.968 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-04T21:51:37.995 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-04T21:51:38.035 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-04T21:51:38.105 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-babel. 2024-09-04T21:51:38.132 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-04T21:51:38.171 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-04T21:51:38.297 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-routes. 2024-09-04T21:51:38.323 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-04T21:51:38.363 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-04T21:51:38.499 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jinja2. 2024-09-04T21:51:38.526 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-04T21:51:38.566 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-04T21:51:38.658 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-04T21:51:38.685 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:38.716 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:38.919 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-04T21:51:38.946 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:38.985 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:39.622 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-04T21:51:39.649 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-04T21:51:39.688 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-04T21:51:40.025 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-routes. 2024-09-04T21:51:40.040 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-04T21:51:40.074 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-04T21:51:40.428 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-04T21:51:40.455 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:40.494 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:41.646 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-04T21:51:41.663 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-04T21:51:41.704 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-04T21:51:42.208 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-joblib. 2024-09-04T21:51:42.235 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-04T21:51:42.274 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-04T21:51:42.652 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-04T21:51:42.680 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-04T21:51:42.719 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-04T21:51:42.988 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn. 2024-09-04T21:51:43.004 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-04T21:51:43.142 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-04T21:51:43.260 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-04T21:51:43.277 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-04T21:51:43.326 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-04T21:51:43.829 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-joblib. 2024-09-04T21:51:43.857 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-04T21:51:43.896 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-04T21:51:43.984 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-04T21:51:44.011 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:44.049 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:44.233 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-04T21:51:44.260 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-04T21:51:44.291 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-04T21:51:44.569 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn. 2024-09-04T21:51:44.596 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-04T21:51:44.721 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-04T21:51:44.994 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cachetools. 2024-09-04T21:51:45.022 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-04T21:51:45.052 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-04T21:51:45.356 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-04T21:51:45.383 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-04T21:51:45.422 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-04T21:51:45.497 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-04T21:51:45.525 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:45.563 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:45.759 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-04T21:51:45.787 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-04T21:51:45.825 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-04T21:51:46.137 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rsa. 2024-09-04T21:51:46.164 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-04T21:51:46.219 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-04T21:51:46.482 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-google-auth. 2024-09-04T21:51:46.498 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-04T21:51:46.540 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-04T21:51:46.550 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cachetools. 2024-09-04T21:51:46.578 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-04T21:51:46.616 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-04T21:51:46.852 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-04T21:51:46.880 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-04T21:51:46.894 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-04T21:51:46.918 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-04T21:51:46.922 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-04T21:51:46.961 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-04T21:51:47.229 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-websocket. 2024-09-04T21:51:47.258 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-04T21:51:47.272 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-04T21:51:47.288 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-04T21:51:47.313 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-04T21:51:47.322 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-04T21:51:47.584 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rsa. 2024-09-04T21:51:47.611 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-04T21:51:47.650 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-04T21:51:47.674 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-04T21:51:47.703 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-04T21:51:47.749 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-04T21:51:47.970 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-google-auth. 2024-09-04T21:51:47.987 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-04T21:51:48.029 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-04T21:51:48.357 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-04T21:51:48.385 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-04T21:51:48.424 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-04T21:51:48.446 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-04T21:51:48.476 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:48.513 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:48.735 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-websocket. 2024-09-04T21:51:48.763 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-04T21:51:48.801 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-04T21:51:48.824 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-04T21:51:48.852 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-04T21:51:48.891 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-04T21:51:49.130 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-04T21:51:49.158 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-04T21:51:49.185 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-04T21:51:49.188 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-04T21:51:49.213 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-04T21:51:49.243 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-04T21:51:49.515 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package jq. 2024-09-04T21:51:49.531 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-04T21:51:49.573 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-04T21:51:49.859 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package socat. 2024-09-04T21:51:49.875 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-04T21:51:49.877 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-04T21:51:49.894 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:49.926 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-04T21:51:49.926 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:50.246 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-04T21:51:50.271 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package xmlstarlet. 2024-09-04T21:51:50.274 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-04T21:51:50.299 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-04T21:51:50.312 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-04T21:51:50.337 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-04T21:51:50.683 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-04T21:51:50.710 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-04T21:51:50.733 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-test. 2024-09-04T21:51:50.749 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-04T21:51:50.761 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:50.808 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-test (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:50.977 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package jq. 2024-09-04T21:51:50.993 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-04T21:51:51.026 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-04T21:51:51.246 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package socat. 2024-09-04T21:51:51.274 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-04T21:51:51.312 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-04T21:51:51.616 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package xmlstarlet. 2024-09-04T21:51:51.644 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-04T21:51:51.682 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-04T21:51:52.012 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-test. 2024-09-04T21:51:52.028 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:52.069 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-test (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:53.329 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-volume. 2024-09-04T21:51:53.357 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:53.395 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-volume (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:53.708 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-04T21:51:53.733 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:53.774 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs-dev (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:54.086 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-04T21:51:54.113 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-04T21:51:54.152 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-04T21:51:54.449 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-04T21:51:54.477 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-04T21:51:54.506 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-04T21:51:54.616 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-volume. 2024-09-04T21:51:54.644 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4703-g3e4a7f4c-1jammy_all.deb ... 2024-09-04T21:51:54.682 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-volume (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:54.877 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package nvme-cli. 2024-09-04T21:51:54.905 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-04T21:51:54.943 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-04T21:51:55.003 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-04T21:51:55.032 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:51:55.069 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs-dev (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:51:55.374 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package pkg-config. 2024-09-04T21:51:55.389 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-04T21:51:55.402 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-04T21:51:55.417 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-04T21:51:55.465 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-04T21:51:55.472 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-04T21:51:55.710 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-04T21:51:55.726 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-04T21:51:55.759 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-04T21:51:55.819 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-04T21:51:55.846 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-04T21:51:55.885 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-04T21:51:56.188 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-04T21:51:56.216 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-04T21:51:56.254 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-04T21:51:56.264 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package nvme-cli. 2024-09-04T21:51:56.292 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-04T21:51:56.330 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-04T21:51:56.515 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastescript. 2024-09-04T21:51:56.532 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-04T21:51:56.565 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-04T21:51:56.803 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package pkg-config. 2024-09-04T21:51:56.830 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-04T21:51:56.869 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pluggy. 2024-09-04T21:51:56.869 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-04T21:51:56.897 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-04T21:51:56.935 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-04T21:51:57.088 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-04T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-04T21:51:57.138 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-04T21:51:57.246 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-psutil. 2024-09-04T21:51:57.275 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-04T21:51:57.313 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-04T21:51:57.542 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-04T21:51:57.570 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-04T21:51:57.608 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-04T21:51:57.649 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-py. 2024-09-04T21:51:57.677 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-04T21:51:57.716 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-04T21:51:57.911 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastescript. 2024-09-04T21:51:57.939 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-04T21:51:57.994 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-04T21:51:58.036 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pygments. 2024-09-04T21:51:58.064 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-04T21:51:58.102 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-04T21:51:58.314 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pluggy. 2024-09-04T21:51:58.342 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-04T21:51:58.380 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-04T21:51:58.581 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-04T21:51:58.609 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-04T21:51:58.648 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-04T21:51:58.667 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-psutil. 2024-09-04T21:51:58.695 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-04T21:51:58.733 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-04T21:51:58.926 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-toml. 2024-09-04T21:51:58.954 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-04T21:51:58.984 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-04T21:51:59.103 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-py. 2024-09-04T21:51:59.131 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-04T21:51:59.178 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-04T21:51:59.237 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pytest. 2024-09-04T21:51:59.265 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-04T21:51:59.303 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-04T21:51:59.506 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pygments. 2024-09-04T21:51:59.534 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-04T21:51:59.572 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-04T21:51:59.631 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplejson. 2024-09-04T21:51:59.648 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-04T21:51:59.681 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-04T21:51:59.934 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-04T21:51:59.950 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-04T21:51:59.984 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-04T21:52:00.069 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-04T21:52:00.097 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-04T21:52:00.135 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-04T21:52:00.454 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-toml. 2024-09-04T21:52:00.482 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-04T21:52:00.521 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-04T21:52:00.721 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package radosgw. 2024-09-04T21:52:00.749 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../117-radosgw_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:52:00.788 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking radosgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:00.849 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pytest. 2024-09-04T21:52:00.877 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-04T21:52:00.915 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-04T21:52:01.252 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplejson. 2024-09-04T21:52:01.280 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-04T21:52:01.318 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-04T21:52:01.580 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-04T21:52:01.596 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-04T21:52:01.629 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-04T21:52:01.674 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package rbd-fuse. 2024-09-04T21:52:01.690 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:52:01.732 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking rbd-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:02.036 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-04T21:52:02.279 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-04T21:52:02.384 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package radosgw. 2024-09-04T21:52:02.405 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-04T21:52:02.413 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../117-radosgw_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:52:02.450 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking radosgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:02.664 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-04T21:52:02.982 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-04T21:52:03.395 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-fuse. 2024-09-04T21:52:03.408 INFO:teuthology.orchestra.run.smithi157.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-04T21:52:03.411 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4703-g3e4a7f4c-1jammy_amd64.deb ... 2024-09-04T21:52:03.453 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:03.473 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:03.741 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-04T21:52:03.998 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user cephadm....done 2024-09-04T21:52:04.000 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-04T21:52:04.126 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-04T21:52:04.129 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-04T21:52:04.363 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-04T21:52:04.389 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-04T21:52:04.690 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-04T21:52:04.719 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-04T21:52:04.841 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-04T21:52:05.141 INFO:teuthology.orchestra.run.smithi112.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-04T21:52:05.142 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-04T21:52:05.211 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:05.410 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-04T21:52:05.536 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-04T21:52:05.677 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user cephadm....done 2024-09-04T21:52:05.838 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-04T21:52:05.892 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-04T21:52:06.185 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-04T21:52:06.190 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-04T21:52:06.436 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-04T21:52:06.491 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-04T21:52:06.596 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-04T21:52:06.759 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-04T21:52:06.847 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-04T21:52:07.019 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-04T21:52:07.115 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-04T21:52:07.266 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-04T21:52:07.496 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-04T21:52:07.584 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-04T21:52:07.921 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-04T21:52:07.936 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-04T21:52:08.196 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-04T21:52:08.197 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-argparse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:08.474 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-04T21:52:08.497 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-04T21:52:08.625 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-04T21:52:08.776 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-04T21:52:08.840 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-04T21:52:08.902 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-04T21:52:09.069 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-04T21:52:09.267 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-04T21:52:09.312 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-04T21:52:09.576 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-04T21:52:09.677 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-04T21:52:09.709 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-04T21:52:09.774 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-04T21:52:09.852 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-argparse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:09.900 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-04T21:52:10.301 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-04T21:52:10.328 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-04T21:52:10.427 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-04T21:52:10.479 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-04T21:52:10.630 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-04T21:52:10.686 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-04T21:52:10.782 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-04T21:52:10.933 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-04T21:52:10.937 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-04T21:52:11.192 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-04T21:52:11.222 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up zip (3.0-12build2) ... 2024-09-04T21:52:11.340 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-04T21:52:11.591 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-04T21:52:11.623 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-04T21:52:11.703 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-04T21:52:11.855 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-04T21:52:12.042 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-04T21:52:12.231 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-04T21:52:12.301 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-04T21:52:12.391 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-04T21:52:12.427 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-04T21:52:12.553 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-04T21:52:12.675 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-04T21:52:12.846 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-04T21:52:12.943 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-04T21:52:13.214 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-04T21:52:13.261 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up zip (3.0-12build2) ... 2024-09-04T21:52:13.412 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-04T21:52:13.465 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-04T21:52:13.866 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-04T21:52:14.130 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-04T21:52:14.159 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-04T21:52:14.424 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-04T21:52:14.510 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-04T21:52:14.558 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-04T21:52:14.684 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-04T21:52:14.753 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-04T21:52:14.854 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:14.977 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-04T21:52:15.172 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-04T21:52:15.386 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-04T21:52:15.515 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-04T21:52:15.662 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-04T21:52:16.072 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-04T21:52:16.416 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-04T21:52:16.713 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-04T21:52:16.825 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-04T21:52:16.973 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:52:17.085 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-04T21:52:17.107 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-04T21:52:17.227 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:17.367 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-04T21:52:17.493 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-04T21:52:17.554 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-04T21:52:17.610 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-04T21:52:17.895 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-04T21:52:17.913 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-04T21:52:18.146 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-04T21:52:18.389 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:52:18.515 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-04T21:52:18.816 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-04T21:52:19.053 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-04T21:52:19.067 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-04T21:52:19.296 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:52:19.318 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-04T21:52:19.456 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-04T21:52:19.561 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-04T21:52:19.740 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-04T21:52:19.812 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-04T21:52:19.916 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-04T21:52:20.063 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-04T21:52:20.067 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-04T21:52:20.189 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-04T21:52:20.377 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-04T21:52:20.466 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-04T21:52:20.592 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-04T21:52:20.636 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-04T21:52:20.884 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-04T21:52:20.904 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:52:21.080 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-04T21:52:21.177 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-04T21:52:21.408 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-04T21:52:21.436 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-04T21:52:21.546 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-04T21:52:21.680 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-04T21:52:21.683 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-04T21:52:21.769 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-04T21:52:21.951 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-04T21:52:22.007 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-04T21:52:22.133 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-04T21:52:22.235 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-04T21:52:22.509 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-04T21:52:22.511 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-04T21:52:22.760 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-04T21:52:22.779 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-04T21:52:22.930 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-04T21:52:23.161 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-04T21:52:23.240 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-04T21:52:23.391 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-04T21:52:23.395 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:52:23.521 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-04T21:52:23.701 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-04T21:52:23.831 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-04T21:52:24.043 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-04T21:52:24.124 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-04T21:52:24.320 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-04T21:52:24.400 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-04T21:52:24.446 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-04T21:52:24.526 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-04T21:52:24.605 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-04T21:52:24.711 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-04T21:52:24.802 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-04T21:52:24.982 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-04T21:52:25.116 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-04T21:52:25.139 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-04T21:52:25.407 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-04T21:52:25.509 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-04T21:52:25.777 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-04T21:52:25.783 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-04T21:52:25.928 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-04T21:52:26.077 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-04T21:52:26.238 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T21:52:26.389 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-04T21:52:26.420 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-04T21:52:26.522 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librados2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:26.656 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:26.765 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-04T21:52:26.782 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-04T21:52:27.108 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-04T21:52:27.118 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libradosstriper1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:27.346 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-04T21:52:27.401 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-04T21:52:27.535 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-04T21:52:27.614 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librbd1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:27.748 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-modules-core (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:27.811 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-04T21:52:27.874 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:28.087 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-04T21:52:28.088 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-04T21:52:28.198 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-04T21:52:28.491 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-04T21:52:28.576 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs-dev (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:28.718 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rados (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:28.845 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librgw2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:28.877 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-04T21:52:28.971 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-04T21:52:29.169 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-04T21:52:29.522 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-04T21:52:29.673 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librados2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:29.851 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:29.985 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-04T21:52:30.134 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rbd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.260 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up rbd-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.330 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.361 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.461 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cephfs (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.481 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libradosstriper1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.571 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:30.632 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-04T21:52:30.925 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librbd1 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:31.052 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-modules-core (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:31.115 INFO:teuthology.orchestra.run.smithi157.stdout:Adding group ceph....done 2024-09-04T21:52:31.203 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:31.448 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-04T21:52:31.448 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-04T21:52:31.466 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user ceph....done 2024-09-04T21:52:31.608 INFO:teuthology.orchestra.run.smithi157.stdout:Setting system user ceph properties....done 2024-09-04T21:52:31.623 INFO:teuthology.orchestra.run.smithi157.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-04T21:52:31.760 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-04T21:52:31.921 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs-dev (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:32.072 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rados (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:32.112 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-04T21:52:32.223 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librgw2 (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:32.374 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-04T21:52:32.572 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-test (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:32.698 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up radosgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:33.145 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-04T21:52:33.145 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-04T21:52:33.567 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-base (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:33.594 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rbd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:33.745 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-fuse (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:33.896 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:33.910 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-04T21:52:34.047 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cephfs (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:34.198 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-common (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:34.385 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mds (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:34.596 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-04T21:52:34.596 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-04T21:52:34.758 INFO:teuthology.orchestra.run.smithi112.stdout:Adding group ceph....done 2024-09-04T21:52:35.062 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:35.202 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user ceph....done 2024-09-04T21:52:35.282 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-04T21:52:35.282 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-04T21:52:35.413 INFO:teuthology.orchestra.run.smithi112.stdout:Setting system user ceph properties....done 2024-09-04T21:52:35.429 INFO:teuthology.orchestra.run.smithi112.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-04T21:52:35.565 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-04T21:52:35.746 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-osd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:35.876 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-04T21:52:36.019 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-04T21:52:36.019 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-04T21:52:36.334 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-test (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:36.473 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:36.493 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up radosgw (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:36.599 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:36.750 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mon (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:36.949 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-04T21:52:36.950 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-04T21:52:36.965 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-04T21:52:36.965 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-04T21:52:37.379 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-base (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:37.434 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:37.569 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:37.695 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-dashboard (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:37.752 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-04T21:52:37.829 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-volume (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:38.207 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-04T21:52:38.222 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mds (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:38.308 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-04T21:52:38.463 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-04T21:52:38.463 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-04T21:52:38.638 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-04T21:52:38.924 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:39.144 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-04T21:52:39.145 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-04T21:52:39.609 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-osd (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:39.913 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-04T21:52:39.914 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-04T21:52:40.012 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-04T21:52:40.386 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:40.537 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:40.645 INFO:teuthology.orchestra.run.smithi157.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-04T21:52:40.651 DEBUG:teuthology.parallel:result is None 2024-09-04T21:52:40.730 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mon (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:40.956 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-04T21:52:40.956 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-04T21:52:41.405 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-cephadm (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:41.565 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:41.666 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-dashboard (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:41.943 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-volume (19.3.0-4703-g3e4a7f4c-1jammy) ... 2024-09-04T21:52:42.504 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-04T21:52:42.630 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-04T21:52:42.816 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-04T21:52:43.850 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-04T21:52:44.518 INFO:teuthology.orchestra.run.smithi112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-04T21:52:44.523 DEBUG:teuthology.parallel:result is None 2024-09-04T21:52:44.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab 2024-09-04T21:52:44.674 DEBUG:teuthology.orchestra.run.smithi112:> dpkg-query -W -f '${Version}' ceph 2024-09-04T21:52:44.692 INFO:teuthology.orchestra.run.smithi112.stdout:19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:52:44.694 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:52:44.694 INFO:teuthology.task.install:The correct ceph version 19.3.0-4703-g3e4a7f4c-1jammy is installed. 2024-09-04T21:52:44.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab 2024-09-04T21:52:44.846 DEBUG:teuthology.orchestra.run.smithi157:> dpkg-query -W -f '${Version}' ceph 2024-09-04T21:52:44.865 INFO:teuthology.orchestra.run.smithi157.stdout:19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:52:44.865 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4703-g3e4a7f4c-1jammy 2024-09-04T21:52:44.865 INFO:teuthology.task.install:The correct ceph version 19.3.0-4703-g3e4a7f4c-1jammy is installed. 2024-09-04T21:52:44.867 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-04T21:52:44.867 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:44.867 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-04T21:52:44.882 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:44.882 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-04T21:52:44.920 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-04T21:52:44.920 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:44.920 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-09-04T21:52:44.936 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-04T21:52:44.995 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:44.995 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-09-04T21:52:45.009 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-04T21:52:45.062 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-04T21:52:45.063 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:45.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-04T21:52:45.076 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-04T21:52:45.132 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:45.132 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-04T21:52:45.148 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-04T21:52:45.202 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-04T21:52:45.203 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:45.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-09-04T21:52:45.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-04T21:52:45.268 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:45.268 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-09-04T21:52:45.282 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-04T21:52:45.334 INFO:teuthology.run_tasks:Running task ceph... 2024-09-04T21:52:45.421 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-04T21:52:45.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2024-09-04T21:52:45.424 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-09-04T21:52:45.436 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-04T21:52:45.437 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-04T21:52:45.478 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-04T21:52:45.491 INFO:tasks.ceph:Creating extra log directories... 2024-09-04T21:52:45.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-04T21:52:45.530 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-04T21:52:45.556 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-04T21:52:45.557 INFO:tasks.ceph:config {'conf': {'client': {'rbd cache': False}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-04T21:52:45.557 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472', 'branch': 'wip-dis-testing', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7889472', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'rbd cache': False}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'live-migration-nbd', 'sha1': '250c2b7303ac79643ee967bc9c5b14e5a0dbbf52'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6930, 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab', 'sleep_before_teardown': 0, 'subset': '4/100', 'suite': 'rbd', 'suite_branch': 'live-migration-nbd', 'suite_path': '/home/teuthworker/src/github.com_effi-ofer_ceph_250c2b7303ac79643ee967bc9c5b14e5a0dbbf52/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/effi-ofer/ceph', 'suite_sha1': '250c2b7303ac79643ee967bc9c5b14e5a0dbbf52', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFoBeeVfhMhHR6V9cArsNl6U3ls8t74E+zhkX0tWvC78u1HGP0Ot1Q671DhxNoswYHbH9sUdt/jiyog/NppFqg=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBORv0KlzsNxY52E2puIKF/wdLgc8vSAwRXY36mQ8K6yQR9oeqsX6NMIlzLz2D7Hc5wjxM90wWQ0pI3q6pssoEek='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-04_17:03:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-04T21:52:45.557 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-04T21:52:45.586 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-04T21:52:45.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-09-04T21:52:45.634 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-09-04T21:52:45.650 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:45.837 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-09-04T21:52:45.843 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-04T21:52:45.843 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-09-04T21:52:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-04 21:47:07.652774337 +0000 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-04 21:47:07.400776582 +0000 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-04 21:47:07.400776582 +0000 2024-09-04T21:52:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-04T21:52:45.890 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-04T21:52:45.944 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-04T21:52:45.944 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-04T21:52:45.944 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000450096 s, 1.1 MB/s 2024-09-04T21:52:45.945 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-04T21:52:45.994 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-09-04T21:52:46.041 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-04T21:52:46.041 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.041 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-04T21:52:46.041 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.041 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-04 21:47:08.116770202 +0000 2024-09-04T21:52:46.042 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-04 21:47:07.876772341 +0000 2024-09-04T21:52:46.042 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-04 21:47:07.876772341 +0000 2024-09-04T21:52:46.042 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-04T21:52:46.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-04T21:52:46.095 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-04T21:52:46.095 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-04T21:52:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000405755 s, 1.3 MB/s 2024-09-04T21:52:46.097 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-04T21:52:46.146 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-09-04T21:52:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-04T21:52:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.193 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-09-04T21:52:46.193 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.193 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-04 21:47:08.632765605 +0000 2024-09-04T21:52:46.194 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-04 21:47:08.364767992 +0000 2024-09-04T21:52:46.194 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-04 21:47:08.364767992 +0000 2024-09-04T21:52:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-04T21:52:46.194 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-04T21:52:46.247 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-04T21:52:46.247 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-04T21:52:46.247 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000422539 s, 1.2 MB/s 2024-09-04T21:52:46.249 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-04T21:52:46.298 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-04 21:47:08.872763466 +0000 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-04 21:47:08.868763503 +0000 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-04 21:47:08.868763503 +0000 2024-09-04T21:52:46.345 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-04T21:52:46.345 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-04T21:52:46.398 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-04T21:52:46.399 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-04T21:52:46.399 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000378823 s, 1.4 MB/s 2024-09-04T21:52:46.400 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-04T21:52:46.450 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-04T21:52:46.450 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:46.451 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-09-04T21:52:46.456 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-04T21:52:46.456 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-09-04T21:52:46.503 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-04T21:52:46.503 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.503 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-09-04T21:52:46.503 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.504 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-04 21:47:07.445945405 +0000 2024-09-04T21:52:46.504 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-04 21:47:07.441945439 +0000 2024-09-04T21:52:46.504 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-04 21:47:07.441945439 +0000 2024-09-04T21:52:46.504 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-09-04T21:52:46.504 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-04T21:52:46.557 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-04T21:52:46.558 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-04T21:52:46.558 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000278798 s, 1.8 MB/s 2024-09-04T21:52:46.559 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-04T21:52:46.608 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-09-04T21:52:46.655 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-04T21:52:46.655 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.655 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 895 Links: 1 2024-09-04T21:52:46.655 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.656 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-04 21:47:08.177938898 +0000 2024-09-04T21:52:46.656 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-04 21:47:07.901941351 +0000 2024-09-04T21:52:46.656 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-04 21:47:07.901941351 +0000 2024-09-04T21:52:46.656 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-09-04T21:52:46.656 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-04T21:52:46.709 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-04T21:52:46.709 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-04T21:52:46.709 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000292152 s, 1.8 MB/s 2024-09-04T21:52:46.710 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-04T21:52:46.764 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-04 21:47:08.657934630 +0000 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-04 21:47:08.413936800 +0000 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-04 21:47:08.413936800 +0000 2024-09-04T21:52:46.811 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-09-04T21:52:46.811 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-04T21:52:46.865 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-04T21:52:46.865 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-04T21:52:46.865 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000308782 s, 1.7 MB/s 2024-09-04T21:52:46.866 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-04T21:52:46.916 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-09-04T21:52:46.962 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 929 Links: 1 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-04 21:47:09.169930080 +0000 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-04 21:47:08.897932497 +0000 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-04 21:47:08.897932497 +0000 2024-09-04T21:52:46.963 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-09-04T21:52:46.963 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-04T21:52:47.017 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-04T21:52:47.017 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-04T21:52:47.017 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000245441 s, 2.1 MB/s 2024-09-04T21:52:47.018 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-04T21:52:47.068 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-04T21:52:47.069 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-04T21:52:47.069 INFO:tasks.ceph:Generating config... 2024-09-04T21:52:47.070 INFO:tasks.ceph:[client] rbd cache = False 2024-09-04T21:52:47.070 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-04T21:52:47.070 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-04T21:52:47.070 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-04T21:52:47.070 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-04T21:52:47.070 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-04T21:52:47.070 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-04T21:52:47.070 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-04T21:52:47.071 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-04T21:52:47.071 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-04T21:52:47.072 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-04T21:52:47.072 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-04T21:52:47.072 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-04T21:52:47.072 INFO:tasks.ceph:Setting up mon.a... 2024-09-04T21:52:47.072 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-04T21:52:47.106 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.keyring 2024-09-04T21:52:47.115 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-04T21:52:47.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-04T21:52:47.243 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.112'), ('mon.c', '[v2:172.21.15.112:3301,v1:172.21.15.112:6790]'), ('mon.b', '172.21.15.157')] 2024-09-04T21:52:47.244 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.112,[v2:172.21.15.112:3301,v1:172.21.15.112:6790],172.21.15.157', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd cache': False}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-04T21:52:47.244 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:47.245 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-04T21:52:47.289 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.112 --addv c '[v2:172.21.15.112:3301,v1:172.21.15.112:6790]' --add b 172.21.15.157 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:47.376 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:47.376 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid 7c7b20bb-5983-49a4-8b2c-296a55ea8d95 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = quincy 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:fsid 7c7b20bb-5983-49a4-8b2c-296a55ea8d95 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2024-09-04T21:52:47.378192+0000 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-09-04T21:52:47.378192+0000 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 17 (quincy) 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:2: [v2:172.21.15.112:3301/0,v1:172.21.15.112:6790/0] mon.c 2024-09-04T21:52:47.377 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-04T21:52:47.379 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-04T21:52:47.385 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7c7b20bb-5983-49a4-8b2c-296a55ea8d95... 2024-09-04T21:52:47.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-04T21:52:47.430 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-09-04T21:52:47.455 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-09-04T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-09-04T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-04T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-09-04T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-09-04T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-04T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-09-04T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-09-04T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-04T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-04T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-09-04T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-04T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = "172.21.15.112,[v2:172.21.15.112:3301,v1:172.21.15.112:6790],172.21.15.157" 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on pool no app = False 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 5000 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 7c7b20bb-5983-49a4-8b2c-296a55ea8d95 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-09-04T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug trim objects = true 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression algorithm = zstd 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 1/20 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 1/20 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 4/10 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: osd blocked scrub grace period = 3600 2024-09-04T21:52:47.463 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: osd max markdown count = 1000 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: osd shutdown pgref assert = True 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-04T21:52:47.464 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.465 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:52:47.466 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout: rbd cache = False 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.c] 2024-09-04T21:52:47.467 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.b] 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-09-04T21:52:47.468 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-09-04T21:52:47.469 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-09-04T21:52:47.470 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.112,[v2:172.21.15.112:3301,v1:172.21.15.112:6790],172.21.15.157" 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: mon client directed command retry = 5 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on pool no app = False 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: ms inject socket failures = 5000 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 7c7b20bb-5983-49a4-8b2c-296a55ea8d95 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-09-04T21:52:47.471 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.472 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression algorithm = zstd 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression mode = aggressive 2024-09-04T21:52:47.473 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 1/20 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 1/20 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 4/10 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: osd blocked scrub grace period = 3600 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.474 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-09-04T21:52:47.475 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-04T21:52:47.476 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout: rbd cache = False 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-09-04T21:52:47.477 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-09-04T21:52:47.478 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-09-04T21:52:47.479 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-04T21:52:47.479 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-04T21:52:47.563 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-04T21:52:47.563 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:47.563 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-04T21:52:47.608 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:47.609 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-04T21:52:47.657 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:52:47.658 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:47.658 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-04T21:52:47.658 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-04T21:52:47.722 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:52:47.722 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:47.769 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:52:47.769 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:47.769 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-04T21:52:47.769 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-04T21:52:47.794 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:52:47.795 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:47.843 INFO:tasks.ceph:Setting up mon nodes... 2024-09-04T21:52:47.843 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-04T21:52:47.844 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-04T21:52:47.887 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-04T21:52:47.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-04T21:52:47.932 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-04T21:52:47.948 INFO:tasks.ceph:Setting up mds nodes... 2024-09-04T21:52:47.948 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-04T21:52:47.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-04T21:52:47.982 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-04T21:52:47.991 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-04T21:52:48.026 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-04T21:52:48.037 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-04T21:52:48.037 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-04T21:52:48.038 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-04T21:52:48.050 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-04T21:52:48.050 INFO:tasks.ceph:role: osd.0 2024-09-04T21:52:48.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:52:48.050 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-04T21:52:48.112 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:52:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:52:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:52:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:52:48.112 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:52:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:52:48.113 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:52:48.113 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:52:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:52:48.113 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:52:48.587 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-04T21:52:48.589 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-04T21:52:48.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-04T21:52:48.752 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-04T21:52:48.763 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:52:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:52:48.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-04T21:52:48.821 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-04T21:52:48.821 INFO:tasks.ceph:role: osd.1 2024-09-04T21:52:48.821 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:52:48.821 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-04T21:52:48.884 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:52:48.884 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:52:48.884 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:52:48.885 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:52:49.321 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-04T21:52:49.324 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-04T21:52:49.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-04T21:52:49.418 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-04T21:52:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:52:49.429 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:52:49.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-04T21:52:49.483 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-04T21:52:49.483 INFO:tasks.ceph:role: osd.2 2024-09-04T21:52:49.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:52:49.483 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:52:49.544 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:52:49.545 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:52:50.000 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-04T21:52:50.002 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-04T21:52:50.002 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-04T21:52:50.096 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-04T21:52:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:52:50.106 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:52:50.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-04T21:52:50.158 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-04T21:52:50.159 INFO:tasks.ceph:role: osd.3 2024-09-04T21:52:50.159 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:52:50.159 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:52:50.221 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:52:50.222 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:52:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:52:50.222 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:52:50.713 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-04T21:52:50.715 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-04T21:52:50.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-04T21:52:50.876 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-04T21:52:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:52:50.887 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:52:50.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:50.993+0000 7f26b377a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-04T21:52:51.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:51.001+0000 7f26b377a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-04T21:52:51.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:51.001+0000 7f26b377a7c0 -1 bdev(0x562484fc9800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-04T21:52:51.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:51.001+0000 7f26b377a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-04T21:52:54.145 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-04T21:52:54.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:54.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:54.241+0000 7f8af64047c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-04T21:52:54.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:54.241+0000 7f8af64047c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-04T21:52:54.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:54.241+0000 7f8af64047c0 -1 bdev(0x55b003437800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-04T21:52:54.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:54.241+0000 7f8af64047c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-04T21:52:57.437 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-04T21:52:57.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:52:57.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:57.573+0000 7fe5595d27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-04T21:52:57.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:57.573+0000 7fe5595d27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-04T21:52:57.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:57.573+0000 7fe5595d27c0 -1 bdev(0x5558ea8b5800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-04T21:52:57.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:52:57.573+0000 7fe5595d27c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-04T21:53:00.743 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-04T21:53:00.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:00.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:53:00.877+0000 7fe2af82e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-04T21:53:00.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:53:00.877+0000 7fe2af82e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-04T21:53:00.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:53:00.877+0000 7fe2af82e7c0 -1 bdev(0x5613fe41d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-04T21:53:00.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-04T21:53:00.877+0000 7fe2af82e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-04T21:53:04.034 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-04T21:53:04.088 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-04T21:53:04.103 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-04T21:53:04.103 INFO:tasks.ceph:role: osd.4 2024-09-04T21:53:04.103 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:53:04.103 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:53:04.159 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:53:04.160 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:53:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:53:04.160 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:53:04.183 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-04T21:53:04.185 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-04T21:53:04.185 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-04T21:53:04.242 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-04T21:53:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:53:04.296 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:53:04.296 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-04T21:53:04.350 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-04T21:53:04.350 INFO:tasks.ceph:role: osd.5 2024-09-04T21:53:04.350 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:53:04.350 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:53:04.407 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:53:04.408 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:53:04.408 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:53:04.408 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:53:04.430 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-04T21:53:04.432 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-04T21:53:04.432 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-04T21:53:04.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-04T21:53:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:53:04.547 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:53:04.548 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-04T21:53:04.600 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-04T21:53:04.601 INFO:tasks.ceph:role: osd.6 2024-09-04T21:53:04.601 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:53:04.601 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-04T21:53:04.654 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:53:04.655 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:53:04.678 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-04T21:53:04.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-04T21:53:04.680 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-04T21:53:04.742 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-04T21:53:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:53:04.796 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:53:04.796 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-04T21:53:04.848 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-04T21:53:04.848 INFO:tasks.ceph:role: osd.7 2024-09-04T21:53:04.848 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:53:04.849 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-04T21:53:04.902 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-04T21:53:04.903 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-04T21:53:04.903 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-04T21:53:04.903 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-04T21:53:04.927 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-04T21:53:04.929 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-04T21:53:04.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-04T21:53:04.992 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-04T21:53:05.043 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-09-04T21:53:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:53:05.045 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:05.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:05.167+0000 7f87690267c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-04T21:53:05.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:05.167+0000 7f87690267c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-04T21:53:05.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:05.167+0000 7f87690267c0 -1 bdev(0x557deeea7800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-04T21:53:05.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:05.167+0000 7f87690267c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-04T21:53:08.305 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-04T21:53:08.319 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:08.404 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:08.399+0000 7f3a3d6857c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-04T21:53:08.404 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:08.399+0000 7f3a3d6857c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-04T21:53:08.404 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:08.399+0000 7f3a3d6857c0 -1 bdev(0x558d0efcd800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-04T21:53:08.404 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:08.399+0000 7f3a3d6857c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-04T21:53:11.532 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-04T21:53:11.587 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:11.670 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:11.663+0000 7f66877df7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-04T21:53:11.670 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:11.663+0000 7f66877df7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-04T21:53:11.670 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:11.663+0000 7f66877df7c0 -1 bdev(0x55abe2b73800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-04T21:53:11.670 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:11.667+0000 7f66877df7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-04T21:53:14.818 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-04T21:53:14.875 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:14.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:14.951+0000 7f19d0df87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-04T21:53:14.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:14.951+0000 7f19d0df87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-04T21:53:14.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:14.951+0000 7f19d0df87c0 -1 bdev(0x5602ba525800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-04T21:53:14.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-04T21:53:14.951+0000 7f19d0df87c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-04T21:53:18.105 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-04T21:53:18.163 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-04T21:53:18.163 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:18.163 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-04T21:53:18.178 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:18.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-04T21:53:18.231 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:18.231 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-04T21:53:18.292 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:18.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-04T21:53:18.347 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:18.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-04T21:53:18.404 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:18.404 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-04T21:53:18.418 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:18.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-04T21:53:18.473 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:18.473 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-04T21:53:18.525 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:18.526 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-04T21:53:18.582 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:18.582 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-04T21:53:18.640 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:18.640 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-04T21:53:18.646 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:18.646 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-04T21:53:18.682 INFO:tasks.ceph:Adding keys to all mons... 2024-09-04T21:53:18.682 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-04T21:53:18.689 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.y] 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCv1thmJRP9NBAAfyoOIJtJ23pZ4FIVMPApPQ== 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCy1thm3tppOxAAy1xKQ+cPVUgFVcVyPaF61A== 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC21thmnnuFDhAAgE7Z1sXtgOc1t/0A4bv8vw== 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC51thm6tpPIhAA+MPY+y4qScUCcQtDZuOzYw== 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.3] 2024-09-04T21:53:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC81thmw69zNBAArtwkqaajVGR1uFE/5hg7+g== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCv1thmupuZNxAA77q65djEPzPOuYBNWoV3Yw== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.4] 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDB1thmXDBPChAAXhmrw49rmlY1NxiJwkOT0g== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.5] 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDE1thmhqAmGBAAXaF8T827FyF5BPqErNRJkA== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.6] 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDH1thmWHD9JxAABlN4k6ZUJIe2TcFCxr4wjA== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.7] 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDK1thmss0AORAAyYX0jTt3FaUNYU1rgkEv/Q== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCv1thmD2qhOhAAaTC/uiDpod8+0ojn4g/6wQ== 2024-09-04T21:53:18.730 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2024-09-04T21:53:18.731 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCw1thmQciiARAA2jdTI4h3GwaGWa+i4mpGOw== 2024-09-04T21:53:18.736 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCv1thmJRP9NBAAfyoOIJtJ23pZ4FIVMPApPQ== 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCy1thm3tppOxAAy1xKQ+cPVUgFVcVyPaF61A== 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC21thmnnuFDhAAgE7Z1sXtgOc1t/0A4bv8vw== 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC51thm6tpPIhAA+MPY+y4qScUCcQtDZuOzYw== 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC81thmw69zNBAArtwkqaajVGR1uFE/5hg7+g== 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2024-09-04T21:53:18.737 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCv1thmupuZNxAA77q65djEPzPOuYBNWoV3Yw== 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDB1thmXDBPChAAXhmrw49rmlY1NxiJwkOT0g== 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDE1thmhqAmGBAAXaF8T827FyF5BPqErNRJkA== 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDH1thmWHD9JxAABlN4k6ZUJIe2TcFCxr4wjA== 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDK1thmss0AORAAyYX0jTt3FaUNYU1rgkEv/Q== 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCv1thmD2qhOhAAaTC/uiDpod8+0ojn4g/6wQ== 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2024-09-04T21:53:18.738 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCw1thmQciiARAA2jdTI4h3GwaGWa+i4mpGOw== 2024-09-04T21:53:18.739 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-04T21:53:18.777 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-04T21:53:18.831 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:18.833 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:18.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:18.918 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:18.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.001 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-04T21:53:19.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-04T21:53:19.217 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.246 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.300 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.329 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.413 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.497 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-04T21:53:19.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-04T21:53:19.581 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-04T21:53:19.638 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-04T21:53:19.666 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-04T21:53:19.725 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-04T21:53:19.725 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-04T21:53:19.764 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-04T21:53:19.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-04T21:53:19.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-04T21:53:19.921 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-04T21:53:20.008 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-04T21:53:20.063 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-04T21:53:20.078 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-04T21:53:20.175 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-04T21:53:20.186 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:20.188 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-04T21:53:20.235 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-04T21:53:20.235 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-04T21:53:20.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-04T21:53:20.238 INFO:tasks.ceph.mon.a:Started 2024-09-04T21:53:20.239 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-04T21:53:20.239 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-04T21:53:20.242 INFO:tasks.ceph.mon.c:Started 2024-09-04T21:53:20.242 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-04T21:53:20.242 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-04T21:53:20.280 INFO:tasks.ceph.mon.b:Started 2024-09-04T21:53:20.280 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-04T21:53:20.280 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-04T21:53:20.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-04T21:53:20.283 INFO:tasks.ceph.mgr.y:Started 2024-09-04T21:53:20.283 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-04T21:53:20.283 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-04T21:53:20.286 INFO:tasks.ceph.mgr.x:Started 2024-09-04T21:53:20.286 DEBUG:tasks.ceph:set 0 configs 2024-09-04T21:53:20.286 INFO:tasks.ceph:Setting crush tunables to default 2024-09-04T21:53:20.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd crush tunables default 2024-09-04T21:53:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2024-09-04T21:53:25.769 INFO:tasks.ceph:check_enable_crimson: False 2024-09-04T21:53:25.769 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-04T21:53:25.770 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:25.770 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-04T21:53:25.780 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:25.780 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-04T21:53:25.833 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:25.834 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-04T21:53:25.886 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-04T21:53:25.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-04T21:53:25.939 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:25.939 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-04T21:53:25.952 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:25.952 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-04T21:53:26.010 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:26.010 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-04T21:53:26.070 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-04T21:53:26.070 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-04T21:53:26.125 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new d210c577-fc3d-48c0-9acb-67afdd5a485a 0 2024-09-04T21:53:26.389 INFO:teuthology.orchestra.run.smithi157.stdout:0 2024-09-04T21:53:26.407 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 3d1c8d55-e5ac-4787-a725-6895bb2bf3f9 1 2024-09-04T21:53:26.612 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-09-04T21:53:26.630 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 5a67b7d1-ef0a-4935-9af0-01389e12d43e 2 2024-09-04T21:53:26.836 INFO:teuthology.orchestra.run.smithi157.stdout:2 2024-09-04T21:53:26.854 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 0b3ec943-bdb7-47e3-bcca-b52070713e9c 3 2024-09-04T21:53:27.059 INFO:teuthology.orchestra.run.smithi157.stdout:3 2024-09-04T21:53:27.076 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 09964dd2-405d-483d-822d-1b8c74497413 4 2024-09-04T21:53:27.274 INFO:teuthology.orchestra.run.smithi157.stdout:4 2024-09-04T21:53:27.291 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 87306896-7b79-487f-ab06-ff337bb72f37 5 2024-09-04T21:53:27.493 INFO:teuthology.orchestra.run.smithi157.stdout:5 2024-09-04T21:53:27.510 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new e7948c3b-e9b4-4a9b-99be-8364f09b8c1f 6 2024-09-04T21:53:27.711 INFO:teuthology.orchestra.run.smithi157.stdout:6 2024-09-04T21:53:27.728 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new c9fdaad2-f75b-4dcd-a170-3e346fda0e16 7 2024-09-04T21:53:27.923 INFO:teuthology.orchestra.run.smithi157.stdout:7 2024-09-04T21:53:27.939 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-04T21:53:27.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-04T21:53:27.941 INFO:tasks.ceph.osd.0:Started 2024-09-04T21:53:27.941 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-04T21:53:27.942 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-04T21:53:27.944 INFO:tasks.ceph.osd.1:Started 2024-09-04T21:53:27.944 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-04T21:53:27.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-04T21:53:27.946 INFO:tasks.ceph.osd.2:Started 2024-09-04T21:53:27.946 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-04T21:53:27.946 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-04T21:53:27.948 INFO:tasks.ceph.osd.3:Started 2024-09-04T21:53:27.948 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-04T21:53:27.949 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-04T21:53:27.951 INFO:tasks.ceph.osd.4:Started 2024-09-04T21:53:27.951 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-04T21:53:27.951 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-04T21:53:27.953 INFO:tasks.ceph.osd.5:Started 2024-09-04T21:53:27.953 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-04T21:53:27.953 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-04T21:53:27.955 INFO:tasks.ceph.osd.6:Started 2024-09-04T21:53:27.955 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-04T21:53:27.955 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-04T21:53:27.957 INFO:tasks.ceph.osd.7:Started 2024-09-04T21:53:27.957 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-04T21:53:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:28.150 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:53:27.919899+0000","last_up_change":"0.000000","last_in_change":"2024-09-04T21:53:27.919899+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:53:28.171 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-04T21:53:28.171 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-04T21:53:28.547 INFO:tasks.ceph.mgr.y.smithi112.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-04T21:53:28.547 INFO:tasks.ceph.mgr.y.smithi112.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-04T21:53:28.548 INFO:tasks.ceph.mgr.y.smithi112.stderr: from numpy import show_config as show_numpy_config 2024-09-04T21:53:28.778 INFO:tasks.ceph.mgr.x.smithi157.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-04T21:53:28.778 INFO:tasks.ceph.mgr.x.smithi157.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-04T21:53:28.778 INFO:tasks.ceph.mgr.x.smithi157.stderr: from numpy import show_config as show_numpy_config 2024-09-04T21:53:28.976 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-04T21:53:29.155 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-04T21:53:29.155 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:53:27.919899+0000","last_up_change":"0.000000","last_in_change":"2024-09-04T21:53:27.919899+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:53:29.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-04T21:53:29.320 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:29.320+0000 7fbec7e267c0 -1 Falling back to public interface 2024-09-04T21:53:29.383 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:29.379+0000 7f9f5f90b7c0 -1 Falling back to public interface 2024-09-04T21:53:29.384 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:29.379+0000 7fd05b6a67c0 -1 Falling back to public interface 2024-09-04T21:53:29.563 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:53:29.559+0000 7f05324317c0 -1 Falling back to public interface 2024-09-04T21:53:29.581 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:29.580+0000 7f6293f9e7c0 -1 Falling back to public interface 2024-09-04T21:53:29.636 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:29.636+0000 7fea37eaa7c0 -1 Falling back to public interface 2024-09-04T21:53:29.800 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:29.795+0000 7fb560e577c0 -1 Falling back to public interface 2024-09-04T21:53:29.809 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:29.808+0000 7f97d564d7c0 -1 Falling back to public interface 2024-09-04T21:53:31.437 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-09-04T21:53:31.436+0000 7fdc5b3b8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-04T21:53:31.437 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-09-04T21:53:31.436+0000 7fdc5b3b8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-04T21:53:33.350 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:33.347+0000 7fd05b6a67c0 -1 osd.5 0 log_to_monitors true 2024-09-04T21:53:33.430 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:33.428+0000 7fbec7e267c0 -1 osd.2 0 log_to_monitors true 2024-09-04T21:53:33.508 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:53:33.503+0000 7f05324317c0 -1 osd.7 0 log_to_monitors true 2024-09-04T21:53:33.622 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:33.620+0000 7f6293f9e7c0 -1 osd.3 0 log_to_monitors true 2024-09-04T21:53:33.680 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:33.675+0000 7f9f5f90b7c0 -1 osd.6 0 log_to_monitors true 2024-09-04T21:53:33.769 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:33.768+0000 7fea37eaa7c0 -1 osd.0 0 log_to_monitors true 2024-09-04T21:53:33.822 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:33.819+0000 7fb560e577c0 -1 osd.4 0 log_to_monitors true 2024-09-04T21:53:34.384 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:34.384+0000 7f97d564d7c0 -1 osd.1 0 log_to_monitors true 2024-09-04T21:53:35.599 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:35.595+0000 7fd057e61640 -1 osd.5 0 waiting for initial osdmap 2024-09-04T21:53:35.613 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:53:35.607+0000 7f052ebec640 -1 osd.7 0 waiting for initial osdmap 2024-09-04T21:53:35.618 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:35.615+0000 7fd04f481640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:35.630 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:53:35.627+0000 7f052620c640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:35.671 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:35.668+0000 7fbec3dcf640 -1 osd.2 0 waiting for initial osdmap 2024-09-04T21:53:35.688 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:35.688+0000 7f6290759640 -1 osd.3 0 waiting for initial osdmap 2024-09-04T21:53:35.692 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:35.692+0000 7fbebbc01640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:35.716 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:35.716+0000 7f6287d79640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:35.977 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-04T21:53:36.388 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-04T21:53:36.388 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":13,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:53:35.684686+0000","last_up_change":"2024-09-04T21:53:35.684686+0000","last_in_change":"2024-09-04T21:53:27.919899+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6817","nonce":3651535617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6819","nonce":3651535617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6823","nonce":3651535617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6821","nonce":3651535617}]},"public_addr":"172.21.15.157:6817/3651535617","cluster_addr":"172.21.15.157:6819/3651535617","heartbeat_back_addr":"172.21.15.157:6823/3651535617","heartbeat_front_addr":"172.21.15.157:6821/3651535617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:53:36.409 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-09-04T21:53:36.653 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:36.652+0000 7fea33e53640 -1 osd.0 0 waiting for initial osdmap 2024-09-04T21:53:36.663 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:36.659+0000 7f9f5b8b4640 -1 osd.6 0 waiting for initial osdmap 2024-09-04T21:53:36.680 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:36.680+0000 7fea2bc85640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:36.691 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:36.687+0000 7f9f536e6640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:36.698 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:36.696+0000 7f97d15f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-04T21:53:36.706 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:36.703+0000 7fb55d612640 -1 osd.4 0 waiting for initial osdmap 2024-09-04T21:53:36.724 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:36.724+0000 7f97c9428640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:36.726 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:36.723+0000 7fb554c32640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:53:43.215 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-04T21:53:43.521 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-04T21:53:43.521 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":17,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:53:39.720616+0000","last_up_change":"2024-09-04T21:53:37.706601+0000","last_in_change":"2024-09-04T21:53:27.919899+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-04T21:53:36.459454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6817","nonce":3808224845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6819","nonce":3808224845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6823","nonce":3808224845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6821","nonce":3808224845}]},"public_addr":"172.21.15.112:6817/3808224845","cluster_addr":"172.21.15.112:6819/3808224845","heartbeat_back_addr":"172.21.15.112:6823/3808224845","heartbeat_front_addr":"172.21.15.112:6821/3808224845","state":["exists","up"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6825","nonce":245081828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6827","nonce":245081828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6831","nonce":245081828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6829","nonce":245081828}]},"public_addr":"172.21.15.112:6825/245081828","cluster_addr":"172.21.15.112:6827/245081828","heartbeat_back_addr":"172.21.15.112:6831/245081828","heartbeat_front_addr":"172.21.15.112:6829/245081828","state":["exists","up"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6801","nonce":1590148387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6803","nonce":1590148387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6807","nonce":1590148387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6805","nonce":1590148387}]},"public_addr":"172.21.15.112:6801/1590148387","cluster_addr":"172.21.15.112:6803/1590148387","heartbeat_back_addr":"172.21.15.112:6807/1590148387","heartbeat_front_addr":"172.21.15.112:6805/1590148387","state":["exists","up"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6809","nonce":2461796488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6811","nonce":2461796488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6815","nonce":2461796488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6813","nonce":2461796488}]},"public_addr":"172.21.15.112:6809/2461796488","cluster_addr":"172.21.15.112:6811/2461796488","heartbeat_back_addr":"172.21.15.112:6815/2461796488","heartbeat_front_addr":"172.21.15.112:6813/2461796488","state":["exists","up"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6825","nonce":3713217432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6827","nonce":3713217432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6831","nonce":3713217432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6829","nonce":3713217432}]},"public_addr":"172.21.15.157:6825/3713217432","cluster_addr":"172.21.15.157:6827/3713217432","heartbeat_back_addr":"172.21.15.157:6831/3713217432","heartbeat_front_addr":"172.21.15.157:6829/3713217432","state":["exists","up"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6801","nonce":3465983375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6803","nonce":3465983375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6808","nonce":3465983375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6805","nonce":3465983375}]},"public_addr":"172.21.15.157:6801/3465983375","cluster_addr":"172.21.15.157:6803/3465983375","heartbeat_back_addr":"172.21.15.157:6808/3465983375","heartbeat_front_addr":"172.21.15.157:6805/3465983375","state":["exists","up"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6817","nonce":3651535617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6819","nonce":3651535617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6823","nonce":3651535617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6821","nonce":3651535617}]},"public_addr":"172.21.15.157:6817/3651535617","cluster_addr":"172.21.15.157:6819/3651535617","heartbeat_back_addr":"172.21.15.157:6823/3651535617","heartbeat_front_addr":"172.21.15.157:6821/3651535617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.725388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:35.420675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.389632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.611983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.796105+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.383432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.731214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.469040+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:53:43.539 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-04T21:53:43.539 INFO:tasks.ceph:Creating RBD pool 2024-09-04T21:53:43.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-04T21:53:44.771 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2024-09-04T21:53:44.802 DEBUG:teuthology.orchestra.run.smithi112:> rbd --cluster ceph pool init rbd 2024-09-04T21:53:47.928 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-04T21:53:47.929 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-04T21:53:47.929 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-04T21:53:48.247 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:48.266 INFO:teuthology.orchestra.run.smithi112.stdout:[{"version":9,"timestamp":"2024-09-04T21:53:36.711286+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"5667.025528"}]}] 2024-09-04T21:53:48.266 INFO:tasks.ceph_manager:config epoch is 9 2024-09-04T21:53:48.266 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-04T21:53:48.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-04T21:53:48.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-04T21:53:48.625 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:48.642 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1247785974},{"type":"v1","addr":"172.21.15.112:6833","nonce":1247785974}]},"active_addr":"172.21.15.112:6833/1247785974","active_change":"2024-09-04T21:53:30.412059+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4110,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":4096537535}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":4126961363}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":1082391471}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":3756046071}]}]} 2024-09-04T21:53:48.646 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-04T21:53:48.647 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-04T21:53:48.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-04T21:53:48.956 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:48.956 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:53:47.793000+0000","last_up_change":"2024-09-04T21:53:37.706601+0000","last_in_change":"2024-09-04T21:53:27.919899+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-04T21:53:36.459454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-04T21:53:43.849962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6817","nonce":3808224845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6819","nonce":3808224845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6823","nonce":3808224845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6821","nonce":3808224845}]},"public_addr":"172.21.15.112:6817/3808224845","cluster_addr":"172.21.15.112:6819/3808224845","heartbeat_back_addr":"172.21.15.112:6823/3808224845","heartbeat_front_addr":"172.21.15.112:6821/3808224845","state":["exists","up"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6825","nonce":245081828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6827","nonce":245081828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6831","nonce":245081828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6829","nonce":245081828}]},"public_addr":"172.21.15.112:6825/245081828","cluster_addr":"172.21.15.112:6827/245081828","heartbeat_back_addr":"172.21.15.112:6831/245081828","heartbeat_front_addr":"172.21.15.112:6829/245081828","state":["exists","up"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6801","nonce":1590148387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6803","nonce":1590148387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6807","nonce":1590148387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6805","nonce":1590148387}]},"public_addr":"172.21.15.112:6801/1590148387","cluster_addr":"172.21.15.112:6803/1590148387","heartbeat_back_addr":"172.21.15.112:6807/1590148387","heartbeat_front_addr":"172.21.15.112:6805/1590148387","state":["exists","up"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6809","nonce":2461796488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6811","nonce":2461796488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6815","nonce":2461796488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6813","nonce":2461796488}]},"public_addr":"172.21.15.112:6809/2461796488","cluster_addr":"172.21.15.112:6811/2461796488","heartbeat_back_addr":"172.21.15.112:6815/2461796488","heartbeat_front_addr":"172.21.15.112:6813/2461796488","state":["exists","up"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6825","nonce":3713217432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6827","nonce":3713217432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6831","nonce":3713217432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6829","nonce":3713217432}]},"public_addr":"172.21.15.157:6825/3713217432","cluster_addr":"172.21.15.157:6827/3713217432","heartbeat_back_addr":"172.21.15.157:6831/3713217432","heartbeat_front_addr":"172.21.15.157:6829/3713217432","state":["exists","up"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6801","nonce":3465983375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6803","nonce":3465983375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6808","nonce":3465983375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6805","nonce":3465983375}]},"public_addr":"172.21.15.157:6801/3465983375","cluster_addr":"172.21.15.157:6803/3465983375","heartbeat_back_addr":"172.21.15.157:6808/3465983375","heartbeat_front_addr":"172.21.15.157:6805/3465983375","state":["exists","up"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6817","nonce":3651535617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6819","nonce":3651535617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6823","nonce":3651535617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6821","nonce":3651535617}]},"public_addr":"172.21.15.157:6817/3651535617","cluster_addr":"172.21.15.157:6819/3651535617","heartbeat_back_addr":"172.21.15.157:6823/3651535617","heartbeat_front_addr":"172.21.15.157:6821/3651535617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.725388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:35.420675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.389632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.611983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.796105+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.383432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.731214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.469040+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:53:48.974 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-04T21:53:48.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-04T21:53:49.274 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:49.275 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:53:47.793000+0000","last_up_change":"2024-09-04T21:53:37.706601+0000","last_in_change":"2024-09-04T21:53:27.919899+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-04T21:53:36.459454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-04T21:53:43.849962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6817","nonce":3808224845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6819","nonce":3808224845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6823","nonce":3808224845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6821","nonce":3808224845}]},"public_addr":"172.21.15.112:6817/3808224845","cluster_addr":"172.21.15.112:6819/3808224845","heartbeat_back_addr":"172.21.15.112:6823/3808224845","heartbeat_front_addr":"172.21.15.112:6821/3808224845","state":["exists","up"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6825","nonce":245081828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6827","nonce":245081828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6831","nonce":245081828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6829","nonce":245081828}]},"public_addr":"172.21.15.112:6825/245081828","cluster_addr":"172.21.15.112:6827/245081828","heartbeat_back_addr":"172.21.15.112:6831/245081828","heartbeat_front_addr":"172.21.15.112:6829/245081828","state":["exists","up"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6801","nonce":1590148387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6803","nonce":1590148387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6807","nonce":1590148387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6805","nonce":1590148387}]},"public_addr":"172.21.15.112:6801/1590148387","cluster_addr":"172.21.15.112:6803/1590148387","heartbeat_back_addr":"172.21.15.112:6807/1590148387","heartbeat_front_addr":"172.21.15.112:6805/1590148387","state":["exists","up"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6809","nonce":2461796488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6811","nonce":2461796488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6815","nonce":2461796488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6813","nonce":2461796488}]},"public_addr":"172.21.15.112:6809/2461796488","cluster_addr":"172.21.15.112:6811/2461796488","heartbeat_back_addr":"172.21.15.112:6815/2461796488","heartbeat_front_addr":"172.21.15.112:6813/2461796488","state":["exists","up"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6825","nonce":3713217432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6827","nonce":3713217432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6831","nonce":3713217432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3713217432},{"type":"v1","addr":"172.21.15.157:6829","nonce":3713217432}]},"public_addr":"172.21.15.157:6825/3713217432","cluster_addr":"172.21.15.157:6827/3713217432","heartbeat_back_addr":"172.21.15.157:6831/3713217432","heartbeat_front_addr":"172.21.15.157:6829/3713217432","state":["exists","up"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6801","nonce":3465983375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6803","nonce":3465983375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6808","nonce":3465983375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6805","nonce":3465983375}]},"public_addr":"172.21.15.157:6801/3465983375","cluster_addr":"172.21.15.157:6803/3465983375","heartbeat_back_addr":"172.21.15.157:6808/3465983375","heartbeat_front_addr":"172.21.15.157:6805/3465983375","state":["exists","up"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6817","nonce":3651535617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6819","nonce":3651535617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6823","nonce":3651535617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3651535617},{"type":"v1","addr":"172.21.15.157:6821","nonce":3651535617}]},"public_addr":"172.21.15.157:6817/3651535617","cluster_addr":"172.21.15.157:6819/3651535617","heartbeat_back_addr":"172.21.15.157:6823/3651535617","heartbeat_front_addr":"172.21.15.157:6821/3651535617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.725388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:35.420675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.389632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.611983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.796105+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.383432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.731214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.469040+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:53:49.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-04T21:53:49.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-04T21:53:49.293 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-04T21:53:49.293 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-04T21:53:49.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-04T21:53:49.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-04T21:53:49.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-04T21:53:49.295 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-04T21:53:49.547 INFO:teuthology.orchestra.run.smithi112.stdout:60129542149 2024-09-04T21:53:49.547 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-04T21:53:49.552 INFO:teuthology.orchestra.run.smithi112.stdout:64424509447 2024-09-04T21:53:49.552 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-04T21:53:49.562 INFO:teuthology.orchestra.run.smithi112.stdout:64424509447 2024-09-04T21:53:49.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-04T21:53:49.563 INFO:teuthology.orchestra.run.smithi112.stdout:60129542150 2024-09-04T21:53:49.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-04T21:53:49.564 INFO:teuthology.orchestra.run.smithi112.stdout:55834574853 2024-09-04T21:53:49.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-04T21:53:49.566 INFO:teuthology.orchestra.run.smithi112.stdout:64424509446 2024-09-04T21:53:49.567 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-04T21:53:49.572 INFO:teuthology.orchestra.run.smithi112.stdout:55834574853 2024-09-04T21:53:49.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-04T21:53:49.578 INFO:teuthology.orchestra.run.smithi112.stdout:64424509446 2024-09-04T21:53:49.579 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-04T21:53:50.043 INFO:teuthology.orchestra.run.smithi112.stdout:60129542148 2024-09-04T21:53:50.070 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.3 2024-09-04T21:53:50.085 INFO:teuthology.orchestra.run.smithi112.stdout:64424509445 2024-09-04T21:53:50.101 INFO:teuthology.orchestra.run.smithi112.stdout:64424509445 2024-09-04T21:53:50.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.0 2024-09-04T21:53:50.111 INFO:teuthology.orchestra.run.smithi112.stdout:60129542149 2024-09-04T21:53:50.114 INFO:teuthology.orchestra.run.smithi112.stdout:55834574852 2024-09-04T21:53:50.125 INFO:teuthology.orchestra.run.smithi112.stdout:55834574852 2024-09-04T21:53:50.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.4 2024-09-04T21:53:50.131 INFO:teuthology.orchestra.run.smithi112.stdout:64424509444 2024-09-04T21:53:50.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-09-04T21:53:50.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.2 2024-09-04T21:53:50.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2024-09-04T21:53:50.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.1 2024-09-04T21:53:50.155 INFO:teuthology.orchestra.run.smithi112.stdout:64424509444 2024-09-04T21:53:50.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.6 2024-09-04T21:53:51.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-04T21:53:51.110 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-04T21:53:51.131 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-04T21:53:51.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-04T21:53:51.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-04T21:53:51.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-04T21:53:51.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-04T21:53:51.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-04T21:53:51.582 INFO:teuthology.orchestra.run.smithi112.stdout:60129542149 2024-09-04T21:53:51.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.3 2024-09-04T21:53:51.608 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.638 INFO:teuthology.orchestra.run.smithi112.stdout:64424509447 2024-09-04T21:53:51.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2024-09-04T21:53:51.663 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.673 INFO:teuthology.orchestra.run.smithi112.stdout:55834574853 2024-09-04T21:53:51.688 INFO:teuthology.orchestra.run.smithi112.stdout:64424509447 2024-09-04T21:53:51.690 INFO:teuthology.orchestra.run.smithi112.stdout:64424509446 2024-09-04T21:53:51.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2024-09-04T21:53:51.691 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.712 INFO:teuthology.orchestra.run.smithi112.stdout:60129542150 2024-09-04T21:53:51.713 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2024-09-04T21:53:51.713 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.4 2024-09-04T21:53:51.715 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.720 INFO:teuthology.orchestra.run.smithi112.stdout:55834574853 2024-09-04T21:53:51.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2024-09-04T21:53:51.731 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-09-04T21:53:51.739 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.740 INFO:teuthology.orchestra.run.smithi112.stdout:64424509446 2024-09-04T21:53:51.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.6 2024-09-04T21:53:51.757 DEBUG:teuthology.parallel:result is None 2024-09-04T21:53:51.757 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-04T21:53:51.757 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:53:52.073 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:52.074 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:53:52.090 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-04T21:53:50.426506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234644,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740076,"statfs":{"total":773094113280,"available":772853837824,"internally_reserved":0,"allocated":1794048,"data_stored":1755262,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.651546"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540731+0000","last_change":"2024-09-04T21:53:48.540731+0000","last_active":"2024-09-04T21:53:48.540731+0000","last_peered":"2024-09-04T21:53:48.540731+0000","last_clean":"2024-09-04T21:53:48.540731+0000","last_became_active":"2024-09-04T21:53:45.789338+0000","last_became_peered":"2024-09-04T21:53:45.789338+0000","last_unstale":"2024-09-04T21:53:48.540731+0000","last_undegraded":"2024-09-04T21:53:48.540731+0000","last_fullsized":"2024-09-04T21:53:48.540731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:39:52.758887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809753+0000","last_change":"2024-09-04T21:53:47.809753+0000","last_active":"2024-09-04T21:53:47.809753+0000","last_peered":"2024-09-04T21:53:47.809753+0000","last_clean":"2024-09-04T21:53:47.809753+0000","last_became_active":"2024-09-04T21:53:45.790034+0000","last_became_peered":"2024-09-04T21:53:45.790034+0000","last_unstale":"2024-09-04T21:53:47.809753+0000","last_undegraded":"2024-09-04T21:53:47.809753+0000","last_fullsized":"2024-09-04T21:53:47.809753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:02:04.535021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001019666,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540589+0000","last_change":"2024-09-04T21:53:48.541062+0000","last_active":"2024-09-04T21:53:48.540589+0000","last_peered":"2024-09-04T21:53:48.540589+0000","last_clean":"2024-09-04T21:53:48.540589+0000","last_became_active":"2024-09-04T21:53:45.797236+0000","last_became_peered":"2024-09-04T21:53:45.797236+0000","last_unstale":"2024-09-04T21:53:48.540589+0000","last_undegraded":"2024-09-04T21:53:48.540589+0000","last_fullsized":"2024-09-04T21:53:48.540589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:15:56.237824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001091266,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809108+0000","last_change":"2024-09-04T21:53:47.809108+0000","last_active":"2024-09-04T21:53:47.809108+0000","last_peered":"2024-09-04T21:53:47.809108+0000","last_clean":"2024-09-04T21:53:47.809108+0000","last_became_active":"2024-09-04T21:53:45.798184+0000","last_became_peered":"2024-09-04T21:53:45.798184+0000","last_unstale":"2024-09-04T21:53:47.809108+0000","last_undegraded":"2024-09-04T21:53:47.809108+0000","last_fullsized":"2024-09-04T21:53:47.809108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:23:51.756798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011943870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.814793+0000","last_change":"2024-09-04T21:53:47.814793+0000","last_active":"2024-09-04T21:53:47.814793+0000","last_peered":"2024-09-04T21:53:47.814793+0000","last_clean":"2024-09-04T21:53:47.814793+0000","last_became_active":"2024-09-04T21:53:45.791807+0000","last_became_peered":"2024-09-04T21:53:45.791807+0000","last_unstale":"2024-09-04T21:53:47.814793+0000","last_undegraded":"2024-09-04T21:53:47.814793+0000","last_fullsized":"2024-09-04T21:53:47.814793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:51:20.756029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.807566+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:53:47.807566+0000","last_peered":"2024-09-04T21:53:47.807566+0000","last_clean":"2024-09-04T21:53:47.807566+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:53:47.807566+0000","last_undegraded":"2024-09-04T21:53:47.807566+0000","last_fullsized":"2024-09-04T21:53:47.807566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540499+0000","last_change":"2024-09-04T21:53:48.541035+0000","last_active":"2024-09-04T21:53:48.540499+0000","last_peered":"2024-09-04T21:53:48.540499+0000","last_clean":"2024-09-04T21:53:48.540499+0000","last_became_active":"2024-09-04T21:53:45.789119+0000","last_became_peered":"2024-09-04T21:53:45.789119+0000","last_unstale":"2024-09-04T21:53:48.540499+0000","last_undegraded":"2024-09-04T21:53:48.540499+0000","last_fullsized":"2024-09-04T21:53:48.540499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:18:06.405491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010964659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.813271+0000","last_change":"2024-09-04T21:53:47.813802+0000","last_active":"2024-09-04T21:53:47.813271+0000","last_peered":"2024-09-04T21:53:47.813271+0000","last_clean":"2024-09-04T21:53:47.813271+0000","last_became_active":"2024-09-04T21:53:45.789500+0000","last_became_peered":"2024-09-04T21:53:45.789500+0000","last_unstale":"2024-09-04T21:53:47.813271+0000","last_undegraded":"2024-09-04T21:53:47.813271+0000","last_fullsized":"2024-09-04T21:53:47.813271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:04:59.647236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":112,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.539923+0000","last_change":"2024-09-04T21:53:38.874369+0000","last_active":"2024-09-04T21:53:48.539923+0000","last_peered":"2024-09-04T21:53:48.539923+0000","last_clean":"2024-09-04T21:53:48.539923+0000","last_became_active":"2024-09-04T21:53:38.870182+0000","last_became_peered":"2024-09-04T21:53:38.870182+0000","last_unstale":"2024-09-04T21:53:48.539923+0000","last_undegraded":"2024-09-04T21:53:48.539923+0000","last_fullsized":"2024-09-04T21:53:48.539923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:17:44.944120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:53:52.091 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:53:52.365 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:52.366 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:53:52.382 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-04T21:53:50.426506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234644,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740076,"statfs":{"total":773094113280,"available":772853837824,"internally_reserved":0,"allocated":1794048,"data_stored":1755262,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.651546"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540731+0000","last_change":"2024-09-04T21:53:48.540731+0000","last_active":"2024-09-04T21:53:48.540731+0000","last_peered":"2024-09-04T21:53:48.540731+0000","last_clean":"2024-09-04T21:53:48.540731+0000","last_became_active":"2024-09-04T21:53:45.789338+0000","last_became_peered":"2024-09-04T21:53:45.789338+0000","last_unstale":"2024-09-04T21:53:48.540731+0000","last_undegraded":"2024-09-04T21:53:48.540731+0000","last_fullsized":"2024-09-04T21:53:48.540731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:39:52.758887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809753+0000","last_change":"2024-09-04T21:53:47.809753+0000","last_active":"2024-09-04T21:53:47.809753+0000","last_peered":"2024-09-04T21:53:47.809753+0000","last_clean":"2024-09-04T21:53:47.809753+0000","last_became_active":"2024-09-04T21:53:45.790034+0000","last_became_peered":"2024-09-04T21:53:45.790034+0000","last_unstale":"2024-09-04T21:53:47.809753+0000","last_undegraded":"2024-09-04T21:53:47.809753+0000","last_fullsized":"2024-09-04T21:53:47.809753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:02:04.535021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001019666,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540589+0000","last_change":"2024-09-04T21:53:48.541062+0000","last_active":"2024-09-04T21:53:48.540589+0000","last_peered":"2024-09-04T21:53:48.540589+0000","last_clean":"2024-09-04T21:53:48.540589+0000","last_became_active":"2024-09-04T21:53:45.797236+0000","last_became_peered":"2024-09-04T21:53:45.797236+0000","last_unstale":"2024-09-04T21:53:48.540589+0000","last_undegraded":"2024-09-04T21:53:48.540589+0000","last_fullsized":"2024-09-04T21:53:48.540589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:15:56.237824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001091266,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809108+0000","last_change":"2024-09-04T21:53:47.809108+0000","last_active":"2024-09-04T21:53:47.809108+0000","last_peered":"2024-09-04T21:53:47.809108+0000","last_clean":"2024-09-04T21:53:47.809108+0000","last_became_active":"2024-09-04T21:53:45.798184+0000","last_became_peered":"2024-09-04T21:53:45.798184+0000","last_unstale":"2024-09-04T21:53:47.809108+0000","last_undegraded":"2024-09-04T21:53:47.809108+0000","last_fullsized":"2024-09-04T21:53:47.809108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:23:51.756798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011943870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.814793+0000","last_change":"2024-09-04T21:53:47.814793+0000","last_active":"2024-09-04T21:53:47.814793+0000","last_peered":"2024-09-04T21:53:47.814793+0000","last_clean":"2024-09-04T21:53:47.814793+0000","last_became_active":"2024-09-04T21:53:45.791807+0000","last_became_peered":"2024-09-04T21:53:45.791807+0000","last_unstale":"2024-09-04T21:53:47.814793+0000","last_undegraded":"2024-09-04T21:53:47.814793+0000","last_fullsized":"2024-09-04T21:53:47.814793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:51:20.756029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.807566+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:53:47.807566+0000","last_peered":"2024-09-04T21:53:47.807566+0000","last_clean":"2024-09-04T21:53:47.807566+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:53:47.807566+0000","last_undegraded":"2024-09-04T21:53:47.807566+0000","last_fullsized":"2024-09-04T21:53:47.807566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540499+0000","last_change":"2024-09-04T21:53:48.541035+0000","last_active":"2024-09-04T21:53:48.540499+0000","last_peered":"2024-09-04T21:53:48.540499+0000","last_clean":"2024-09-04T21:53:48.540499+0000","last_became_active":"2024-09-04T21:53:45.789119+0000","last_became_peered":"2024-09-04T21:53:45.789119+0000","last_unstale":"2024-09-04T21:53:48.540499+0000","last_undegraded":"2024-09-04T21:53:48.540499+0000","last_fullsized":"2024-09-04T21:53:48.540499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:18:06.405491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010964659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.813271+0000","last_change":"2024-09-04T21:53:47.813802+0000","last_active":"2024-09-04T21:53:47.813271+0000","last_peered":"2024-09-04T21:53:47.813271+0000","last_clean":"2024-09-04T21:53:47.813271+0000","last_became_active":"2024-09-04T21:53:45.789500+0000","last_became_peered":"2024-09-04T21:53:45.789500+0000","last_unstale":"2024-09-04T21:53:47.813271+0000","last_undegraded":"2024-09-04T21:53:47.813271+0000","last_fullsized":"2024-09-04T21:53:47.813271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:04:59.647236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":112,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.539923+0000","last_change":"2024-09-04T21:53:38.874369+0000","last_active":"2024-09-04T21:53:48.539923+0000","last_peered":"2024-09-04T21:53:48.539923+0000","last_clean":"2024-09-04T21:53:48.539923+0000","last_became_active":"2024-09-04T21:53:38.870182+0000","last_became_peered":"2024-09-04T21:53:38.870182+0000","last_unstale":"2024-09-04T21:53:48.539923+0000","last_undegraded":"2024-09-04T21:53:48.539923+0000","last_fullsized":"2024-09-04T21:53:48.539923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:17:44.944120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:53:52.383 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-04T21:53:52.383 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-04T21:53:52.383 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-04T21:53:52.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-04T21:53:52.694 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:52.695 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-04T21:53:52.712 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-04T21:53:52.712 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-04T21:53:52.727 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-04T21:53:52.727 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'rbd cache': False}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3e4a7f4c58c7c6fe936e6fda96ba06459c94d8ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'live-migration-nbd', 'sha1': '250c2b7303ac79643ee967bc9c5b14e5a0dbbf52'}} 2024-09-04T21:53:52.727 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-04T21:53:52.728 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-04T21:53:53.051 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-04T21:53:53.051 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-04T21:53:53.052 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:53:53.331 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:53.331 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:53:53.348 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-04T21:53:52.427075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234644,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740076,"statfs":{"total":773094113280,"available":772853837824,"internally_reserved":0,"allocated":1794048,"data_stored":1755262,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.649721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540731+0000","last_change":"2024-09-04T21:53:48.540731+0000","last_active":"2024-09-04T21:53:48.540731+0000","last_peered":"2024-09-04T21:53:48.540731+0000","last_clean":"2024-09-04T21:53:48.540731+0000","last_became_active":"2024-09-04T21:53:45.789338+0000","last_became_peered":"2024-09-04T21:53:45.789338+0000","last_unstale":"2024-09-04T21:53:48.540731+0000","last_undegraded":"2024-09-04T21:53:48.540731+0000","last_fullsized":"2024-09-04T21:53:48.540731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:39:52.758887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809753+0000","last_change":"2024-09-04T21:53:47.809753+0000","last_active":"2024-09-04T21:53:47.809753+0000","last_peered":"2024-09-04T21:53:47.809753+0000","last_clean":"2024-09-04T21:53:47.809753+0000","last_became_active":"2024-09-04T21:53:45.790034+0000","last_became_peered":"2024-09-04T21:53:45.790034+0000","last_unstale":"2024-09-04T21:53:47.809753+0000","last_undegraded":"2024-09-04T21:53:47.809753+0000","last_fullsized":"2024-09-04T21:53:47.809753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:02:04.535021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001019666,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540589+0000","last_change":"2024-09-04T21:53:48.541062+0000","last_active":"2024-09-04T21:53:48.540589+0000","last_peered":"2024-09-04T21:53:48.540589+0000","last_clean":"2024-09-04T21:53:48.540589+0000","last_became_active":"2024-09-04T21:53:45.797236+0000","last_became_peered":"2024-09-04T21:53:45.797236+0000","last_unstale":"2024-09-04T21:53:48.540589+0000","last_undegraded":"2024-09-04T21:53:48.540589+0000","last_fullsized":"2024-09-04T21:53:48.540589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:15:56.237824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001091266,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809108+0000","last_change":"2024-09-04T21:53:47.809108+0000","last_active":"2024-09-04T21:53:47.809108+0000","last_peered":"2024-09-04T21:53:47.809108+0000","last_clean":"2024-09-04T21:53:47.809108+0000","last_became_active":"2024-09-04T21:53:45.798184+0000","last_became_peered":"2024-09-04T21:53:45.798184+0000","last_unstale":"2024-09-04T21:53:47.809108+0000","last_undegraded":"2024-09-04T21:53:47.809108+0000","last_fullsized":"2024-09-04T21:53:47.809108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:23:51.756798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011943870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.814793+0000","last_change":"2024-09-04T21:53:47.814793+0000","last_active":"2024-09-04T21:53:47.814793+0000","last_peered":"2024-09-04T21:53:47.814793+0000","last_clean":"2024-09-04T21:53:47.814793+0000","last_became_active":"2024-09-04T21:53:45.791807+0000","last_became_peered":"2024-09-04T21:53:45.791807+0000","last_unstale":"2024-09-04T21:53:47.814793+0000","last_undegraded":"2024-09-04T21:53:47.814793+0000","last_fullsized":"2024-09-04T21:53:47.814793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:51:20.756029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.807566+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:53:47.807566+0000","last_peered":"2024-09-04T21:53:47.807566+0000","last_clean":"2024-09-04T21:53:47.807566+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:53:47.807566+0000","last_undegraded":"2024-09-04T21:53:47.807566+0000","last_fullsized":"2024-09-04T21:53:47.807566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540499+0000","last_change":"2024-09-04T21:53:48.541035+0000","last_active":"2024-09-04T21:53:48.540499+0000","last_peered":"2024-09-04T21:53:48.540499+0000","last_clean":"2024-09-04T21:53:48.540499+0000","last_became_active":"2024-09-04T21:53:45.789119+0000","last_became_peered":"2024-09-04T21:53:45.789119+0000","last_unstale":"2024-09-04T21:53:48.540499+0000","last_undegraded":"2024-09-04T21:53:48.540499+0000","last_fullsized":"2024-09-04T21:53:48.540499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:18:06.405491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010964659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.813271+0000","last_change":"2024-09-04T21:53:47.813802+0000","last_active":"2024-09-04T21:53:47.813271+0000","last_peered":"2024-09-04T21:53:47.813271+0000","last_clean":"2024-09-04T21:53:47.813271+0000","last_became_active":"2024-09-04T21:53:45.789500+0000","last_became_peered":"2024-09-04T21:53:45.789500+0000","last_unstale":"2024-09-04T21:53:47.813271+0000","last_undegraded":"2024-09-04T21:53:47.813271+0000","last_fullsized":"2024-09-04T21:53:47.813271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:04:59.647236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":112,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.539923+0000","last_change":"2024-09-04T21:53:38.874369+0000","last_active":"2024-09-04T21:53:48.539923+0000","last_peered":"2024-09-04T21:53:48.539923+0000","last_clean":"2024-09-04T21:53:48.539923+0000","last_became_active":"2024-09-04T21:53:38.870182+0000","last_became_peered":"2024-09-04T21:53:38.870182+0000","last_unstale":"2024-09-04T21:53:48.539923+0000","last_undegraded":"2024-09-04T21:53:48.539923+0000","last_fullsized":"2024-09-04T21:53:48.539923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:17:44.944120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:53:53.349 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:53:53.624 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:53:53.624 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:53:53.640 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-04T21:53:52.427075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234644,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740076,"statfs":{"total":773094113280,"available":772853837824,"internally_reserved":0,"allocated":1794048,"data_stored":1755262,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.649721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540731+0000","last_change":"2024-09-04T21:53:48.540731+0000","last_active":"2024-09-04T21:53:48.540731+0000","last_peered":"2024-09-04T21:53:48.540731+0000","last_clean":"2024-09-04T21:53:48.540731+0000","last_became_active":"2024-09-04T21:53:45.789338+0000","last_became_peered":"2024-09-04T21:53:45.789338+0000","last_unstale":"2024-09-04T21:53:48.540731+0000","last_undegraded":"2024-09-04T21:53:48.540731+0000","last_fullsized":"2024-09-04T21:53:48.540731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:39:52.758887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809753+0000","last_change":"2024-09-04T21:53:47.809753+0000","last_active":"2024-09-04T21:53:47.809753+0000","last_peered":"2024-09-04T21:53:47.809753+0000","last_clean":"2024-09-04T21:53:47.809753+0000","last_became_active":"2024-09-04T21:53:45.790034+0000","last_became_peered":"2024-09-04T21:53:45.790034+0000","last_unstale":"2024-09-04T21:53:47.809753+0000","last_undegraded":"2024-09-04T21:53:47.809753+0000","last_fullsized":"2024-09-04T21:53:47.809753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:02:04.535021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001019666,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540589+0000","last_change":"2024-09-04T21:53:48.541062+0000","last_active":"2024-09-04T21:53:48.540589+0000","last_peered":"2024-09-04T21:53:48.540589+0000","last_clean":"2024-09-04T21:53:48.540589+0000","last_became_active":"2024-09-04T21:53:45.797236+0000","last_became_peered":"2024-09-04T21:53:45.797236+0000","last_unstale":"2024-09-04T21:53:48.540589+0000","last_undegraded":"2024-09-04T21:53:48.540589+0000","last_fullsized":"2024-09-04T21:53:48.540589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:15:56.237824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001091266,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.809108+0000","last_change":"2024-09-04T21:53:47.809108+0000","last_active":"2024-09-04T21:53:47.809108+0000","last_peered":"2024-09-04T21:53:47.809108+0000","last_clean":"2024-09-04T21:53:47.809108+0000","last_became_active":"2024-09-04T21:53:45.798184+0000","last_became_peered":"2024-09-04T21:53:45.798184+0000","last_unstale":"2024-09-04T21:53:47.809108+0000","last_undegraded":"2024-09-04T21:53:47.809108+0000","last_fullsized":"2024-09-04T21:53:47.809108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:23:51.756798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011943870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.814793+0000","last_change":"2024-09-04T21:53:47.814793+0000","last_active":"2024-09-04T21:53:47.814793+0000","last_peered":"2024-09-04T21:53:47.814793+0000","last_clean":"2024-09-04T21:53:47.814793+0000","last_became_active":"2024-09-04T21:53:45.791807+0000","last_became_peered":"2024-09-04T21:53:45.791807+0000","last_unstale":"2024-09-04T21:53:47.814793+0000","last_undegraded":"2024-09-04T21:53:47.814793+0000","last_fullsized":"2024-09-04T21:53:47.814793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:51:20.756029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.807566+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:53:47.807566+0000","last_peered":"2024-09-04T21:53:47.807566+0000","last_clean":"2024-09-04T21:53:47.807566+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:53:47.807566+0000","last_undegraded":"2024-09-04T21:53:47.807566+0000","last_fullsized":"2024-09-04T21:53:47.807566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.540499+0000","last_change":"2024-09-04T21:53:48.541035+0000","last_active":"2024-09-04T21:53:48.540499+0000","last_peered":"2024-09-04T21:53:48.540499+0000","last_clean":"2024-09-04T21:53:48.540499+0000","last_became_active":"2024-09-04T21:53:45.789119+0000","last_became_peered":"2024-09-04T21:53:45.789119+0000","last_unstale":"2024-09-04T21:53:48.540499+0000","last_undegraded":"2024-09-04T21:53:48.540499+0000","last_fullsized":"2024-09-04T21:53:48.540499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:18:06.405491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010964659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:47.813271+0000","last_change":"2024-09-04T21:53:47.813802+0000","last_active":"2024-09-04T21:53:47.813271+0000","last_peered":"2024-09-04T21:53:47.813271+0000","last_clean":"2024-09-04T21:53:47.813271+0000","last_became_active":"2024-09-04T21:53:45.789500+0000","last_became_peered":"2024-09-04T21:53:45.789500+0000","last_unstale":"2024-09-04T21:53:47.813271+0000","last_undegraded":"2024-09-04T21:53:47.813271+0000","last_fullsized":"2024-09-04T21:53:47.813271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:04:59.647236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":112,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-04T21:53:48.539923+0000","last_change":"2024-09-04T21:53:38.874369+0000","last_active":"2024-09-04T21:53:48.539923+0000","last_peered":"2024-09-04T21:53:48.539923+0000","last_clean":"2024-09-04T21:53:48.539923+0000","last_became_active":"2024-09-04T21:53:38.870182+0000","last_became_peered":"2024-09-04T21:53:38.870182+0000","last_unstale":"2024-09-04T21:53:48.539923+0000","last_undegraded":"2024-09-04T21:53:48.539923+0000","last_fullsized":"2024-09-04T21:53:48.539923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:17:44.944120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":662179,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":71830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:53:53.641 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-04T21:53:53.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 21 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:fsid 7c7b20bb-5983-49a4-8b2c-296a55ea8d95 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-09-04T21:53:25.631463+0000 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:modified 2024-09-04T21:53:47.793000+0000 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:crush_version 5 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:full_ratio 0.95 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:backfillfull_ratio 0.9 2024-09-04T21:53:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:nearfull_ratio 0.85 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:require_min_compat_client luminous 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:min_compat_client jewel 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:require_osd_release squid 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:stretch_mode_enabled false 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 21 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.99 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout: removed_snaps_queue [2~1] 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:max_osd 8 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6816/3808224845,v1:172.21.15.112:6817/3808224845] [v2:172.21.15.112:6818/3808224845,v1:172.21.15.112:6819/3808224845] exists,up d210c577-fc3d-48c0-9acb-67afdd5a485a 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6824/245081828,v1:172.21.15.112:6825/245081828] [v2:172.21.15.112:6826/245081828,v1:172.21.15.112:6827/245081828] exists,up 3d1c8d55-e5ac-4787-a725-6895bb2bf3f9 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6800/1590148387,v1:172.21.15.112:6801/1590148387] [v2:172.21.15.112:6802/1590148387,v1:172.21.15.112:6803/1590148387] exists,up 5a67b7d1-ef0a-4935-9af0-01389e12d43e 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6808/2461796488,v1:172.21.15.112:6809/2461796488] [v2:172.21.15.112:6810/2461796488,v1:172.21.15.112:6811/2461796488] exists,up 0b3ec943-bdb7-47e3-bcca-b52070713e9c 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6824/3713217432,v1:172.21.15.157:6825/3713217432] [v2:172.21.15.157:6826/3713217432,v1:172.21.15.157:6827/3713217432] exists,up 09964dd2-405d-483d-822d-1b8c74497413 2024-09-04T21:53:53.931 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6807/352715412,v1:172.21.15.157:6809/352715412] [v2:172.21.15.157:6810/352715412,v1:172.21.15.157:6811/352715412] exists,up 87306896-7b79-487f-ab06-ff337bb72f37 2024-09-04T21:53:53.932 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6800/3465983375,v1:172.21.15.157:6801/3465983375] [v2:172.21.15.157:6802/3465983375,v1:172.21.15.157:6803/3465983375] exists,up e7948c3b-e9b4-4a9b-99be-8364f09b8c1f 2024-09-04T21:53:53.932 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6816/3651535617,v1:172.21.15.157:6817/3651535617] [v2:172.21.15.157:6818/3651535617,v1:172.21.15.157:6819/3651535617] exists,up c9fdaad2-f75b-4dcd-a170-3e346fda0e16 2024-09-04T21:53:53.947 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6816/3808224845,v1:172.21.15.112:6817/3808224845] [v2:172.21.15.112:6818/3808224845,v1:172.21.15.112:6819/3808224845] exists,up d210c577-fc3d-48c0-9acb-67afdd5a485a', 'osd.1 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6824/245081828,v1:172.21.15.112:6825/245081828] [v2:172.21.15.112:6826/245081828,v1:172.21.15.112:6827/245081828] exists,up 3d1c8d55-e5ac-4787-a725-6895bb2bf3f9', 'osd.2 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6800/1590148387,v1:172.21.15.112:6801/1590148387] [v2:172.21.15.112:6802/1590148387,v1:172.21.15.112:6803/1590148387] exists,up 5a67b7d1-ef0a-4935-9af0-01389e12d43e', 'osd.3 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6808/2461796488,v1:172.21.15.112:6809/2461796488] [v2:172.21.15.112:6810/2461796488,v1:172.21.15.112:6811/2461796488] exists,up 0b3ec943-bdb7-47e3-bcca-b52070713e9c', 'osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6824/3713217432,v1:172.21.15.157:6825/3713217432] [v2:172.21.15.157:6826/3713217432,v1:172.21.15.157:6827/3713217432] exists,up 09964dd2-405d-483d-822d-1b8c74497413', 'osd.5 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6807/352715412,v1:172.21.15.157:6809/352715412] [v2:172.21.15.157:6810/352715412,v1:172.21.15.157:6811/352715412] exists,up 87306896-7b79-487f-ab06-ff337bb72f37', 'osd.6 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6800/3465983375,v1:172.21.15.157:6801/3465983375] [v2:172.21.15.157:6802/3465983375,v1:172.21.15.157:6803/3465983375] exists,up e7948c3b-e9b4-4a9b-99be-8364f09b8c1f', 'osd.7 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6816/3651535617,v1:172.21.15.157:6817/3651535617] [v2:172.21.15.157:6818/3651535617,v1:172.21.15.157:6819/3651535617] exists,up c9fdaad2-f75b-4dcd-a170-3e346fda0e16'] 2024-09-04T21:53:53.947 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-04T21:53:54.049 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "mon.a", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster": "ceph", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "admin_socket_mode": "", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_client_required": "cephx, none", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_cluster_required": "cephx", 2024-09-04T21:53:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_debug": "true", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_service_required": "cephx", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "auth_supported": "cephx", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_aio": "true", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_aio_poll_ms": "250", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_aio_reap_max": "16", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_async_discard_threads": "0", 2024-09-04T21:53:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_block_size": "4096", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_debug_aio": "false", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_debug_inflight_ios": "false", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_enable_discard": "false", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_flock_retry": "3", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_inject_crash": "0", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_ioring": "false", 2024-09-04T21:53:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_ioring_hipri": "false", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bdev_type": "", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_alloc_size": "1048576", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_allocator": "hybrid", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_buffered_io": "true", 2024-09-04T21:53:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_check_for_zeros": "false", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_compact_log_sync": "false", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_max_log_runway": "4194304", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_max_prefetch": "1048576", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_min_flush_size": "524288", 2024-09-04T21:53:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_min_log_runway": "1048576", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_replay_recovery": "false", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_sync_write": "false", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_allocation_from_file": "true", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_allocator": "hybrid", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_async_db_compaction": "true", 2024-09-04T21:53:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bdev_label_multi": "true", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_create": "true", 2024-09-04T21:53:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_db_create": "false", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_db_path": "", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_db_size": "0", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_path": "", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_preallocate_file": "false", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_size": "107374182400", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_wal_create": "false", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_wal_path": "", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_block_wal_size": "100663296", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bluefs": "true", 2024-09-04T21:53:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_autotune": "true", 2024-09-04T21:53:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_size": "0", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_type": "2q", 2024-09-04T21:53:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_clone_cow": "true", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_mode": "none", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-04T21:53:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_csum_type": "crc32c", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_freelist": "false", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-04T21:53:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_small_allocations": "0", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_default_buffered_read": "true", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_default_buffered_write": "false", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-04T21:53:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fail_eio": "false", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-04T21:53:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_on_mount": "false", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_on_umount": "false", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-04T21:53:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_ignore_data_csum": "false", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_log_op_age": "5.000000", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_max_alloc_size": "0", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_max_blob_size": "0", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-04T21:53:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_max_deferred_txc": "32", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_min_alloc_size": "0", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_nid_prealloc": "1024", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-04T21:53:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_retry_disk_reads": "3", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_rocksdb_cf": "true", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-04T21:53:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_spdk_mem": "512", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_tracing": "false", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-04T21:53:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_write_v2": "false", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_write_v2_random": "false", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_zero_block_detection": "false", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-04T21:53:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-04T21:53:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_cluster_require_signatures": "false", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_cluster_require_version": "2", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_require_signatures": "false", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_require_version": "2", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_service_require_signatures": "false", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_service_require_version": "2", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "cephx_sign_messages": "true", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "chdir": "", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_acl_type": "", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_asio_thread_count": "2", 2024-09-04T21:53:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_cache_mid": "0.750000", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_cache_size": "16384", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_caps_release_delay": "5", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_check_pool_perm": "true", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_debug_force_sync_read": "false", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_debug_getattr_caps": "false", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_debug_inject_features": "", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_debug_inject_tick_delay": "0", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_die_on_failed_remount": "false", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_dirsize_rbytes": "true", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_force_lazyio": "false", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_fs": "", 2024-09-04T21:53:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_inject_release_failure": "false", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_max_inline_size": "4096", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_mds_namespace": "", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_metadata": "", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_mount_gid": "-1", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_mount_timeout": "300", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_mount_uid": "-1", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_mountpoint": "/", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_notify_timeout": "10", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_oc": "true", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_oc_max_dirty": "104857600", 2024-09-04T21:53:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_oc_max_objects": "1000", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_oc_size": "209715200", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_oc_target_dirty": "8388608", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_permissions": "true", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_quota": "true", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_quota_df": "true", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_readahead_max_bytes": "0", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_readahead_max_periods": "4", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_readahead_min": "131072", 2024-09-04T21:53:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "client_reconnect_stale": "false", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_shutdown_timeout": "30", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_snapdir": ".snap", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_tick_interval": "1", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_trace": "", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_try_dentry_invalidate": "false", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_use_faked_inos": "false", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_use_random_mds": "false", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_graylog": "false", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_graylog_port": "12201", 2024-09-04T21:53:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_monitors": "default=true", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_syslog": "false", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "clog_to_syslog_level": "info", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "-", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_network": "", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_network_interface": "", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "compressor_zlib_isal": "false", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "compressor_zlib_level": "5", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "compressor_zlib_winsize": "-15", 2024-09-04T21:53:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "compressor_zstd_level": "1", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_alien_op_num_threads": "6", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_osd_stat_interval": "0", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_seastar_cpu_cores": "", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crimson_seastar_num_threads": "0", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "", 2024-09-04T21:53:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location_hook": "", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location_hook_timeout": "10", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "daemonize": "false", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "daos_pool": "tank", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_alienstore": "0/5", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_allow_any_pool_priority": "false", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_asok": "1/5", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_asok_assert_abort": "false", 2024-09-04T21:53:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_asserts_on_shutdown": "true", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_auth": "20/20", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_bdev": "1/3", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_bluefs": "1/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_bluestore": "1/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_buffer": "0/1", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_ceph_dedup": "0/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_ceph_exporter": "1/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_cephfs_mirror": "0/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_cephsqlite": "0/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_client": "0/5", 2024-09-04T21:53:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_compressor": "1/5", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_context": "0/1", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_crimson_interrupt": "0/5", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_crush": "1/1", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_crypto": "1/5", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_cyanstore": "0/5", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_deliberately_leak_memory": "false", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_disable_randomized_ping": "false", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_dpdk": "1/5", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_eventtrace": "1/5", 2024-09-04T21:53:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_filer": "0/1", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_filestore": "1/3", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_finisher": "1/1", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_fuse": "1/5", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_heartbeat_testing_span": "0", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_heartbeatmap": "1/5", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_javaclient": "1/5", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_journal": "1/3", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_journaler": "0/5", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_kstore": "1/5", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_lockdep": "0/1", 2024-09-04T21:53:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mclock": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds_balancer": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds_locker": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds_log": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds_log_expire": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds_migrator": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mds_quiesce": "3/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_memstore": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mgr": "2/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mgrc": "1/5", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_mon": "20/20", 2024-09-04T21:53:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_monc": "0/10", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_ms": "1/1", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_none": "0/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_objclass": "0/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_objectcacher": "0/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_objecter": "0/1", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_optracker": "0/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_osd": "1/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_paxos": "20/20", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_perfcounter": "1/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_prioritycache": "1/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rados": "0/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rbd": "0/5", 2024-09-04T21:53:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rbd_mirror": "0/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rbd_pwl": "0/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rbd_replay": "0/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_refs": "0/0", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_reserver": "1/1", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_access": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_datacache": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_dbstore": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_flight": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_notification": "1/5", 2024-09-04T21:53:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rgw_sync": "1/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_rocksdb": "4/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_backref": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_cache": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_cleaner": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_device": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_epm": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_journal": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_lba": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_odata": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_omap": "0/5", 2024-09-04T21:53:54.085 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_onode": "0/5", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_t": "0/5", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_seastore_tm": "0/5", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_striper": "0/1", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_test": "0/5", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_throttle": "1/1", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_timer": "0/1", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_tp": "0/5", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "debug_trace": "1/5", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "device_failure_prediction_mode": "none", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-04T21:53:54.086 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "err_to_graylog": "false", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "err_to_journald": "false", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "err_to_stderr": "true", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "err_to_syslog": "false", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event_tracing": "false", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_addr": "0.0.0.0", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_cert_file": "", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_http_port": "9926", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_key_file": "", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_prio_limit": "5", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-04T21:53:54.087 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_sort_metrics": "true", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "exporter_stats_period": "5", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "fake_statfs_for_testing": "0", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "fatal_signal_handlers": "true", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filer_max_purge_ops": "10", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filer_max_truncate_ops": "128", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_apply_finisher_threads": "1", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_blackhole": "false", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_btrfs_clone_range": "true", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_btrfs_snap": "true", 2024-09-04T21:53:54.088 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_caller_concurrency": "10", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_collect_device_partition_information": "true", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_commit_timeout": "600.000000", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_debug_inject_read_err": "false", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_debug_omap_check": "false", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_debug_verify_split": "false", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_dump_file": "", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fadvise": "true", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fail_eio": "true", 2024-09-04T21:53:54.089 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fd_cache_shards": "16", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fd_cache_size": "128", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fiemap": "false", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fiemap_threshold": "4096", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_inject_stall": "0", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_journal_parallel": "false", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_journal_trailing": "false", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_journal_writeahead": "false", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_kill_at": "0", 2024-09-04T21:53:54.090 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattrs": "0", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_xattr_value_size": "0", 2024-09-04T21:53:54.091 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_merge_threshold": "-10", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_odsync_write": "false", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_omap_backend": "rocksdb", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_omap_backend_path": "", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-04T21:53:54.092 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_op_thread_timeout": "60", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_op_threads": "2", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_punch_hole": "false", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-04T21:53:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_queue_max_ops": "50", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_seek_data_hole": "false", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_sloppy_crc": "false", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_splice": "false", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_split_multiple": "2", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_split_rand_factor": "20", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_update_to": "1000", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-04T21:53:54.094 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_enable": "true", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-04T21:53:54.095 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_xfs_extsize": "false", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "filestore_zfs_snap": "false", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fio_dir": "/tmp/fio", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "7c7b20bb-5983-49a4-8b2c-296a55ea8d95", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_allow_other": "true", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_atomic_o_trunc": "true", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_big_writes": "true", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_debug": "false", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_default_permissions": "false", 2024-09-04T21:53:54.096 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_disable_pagecache": "false", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_max_write": "0", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_multithreaded": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_require_active_mds": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_set_user_groups": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_splice_move": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_splice_read": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_splice_write": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "fuse_use_invalidate_cb": "true", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-04T21:53:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: "gss_target_name": "ceph", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_file": "", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_inject_failure": "0", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_interval": "5", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "host": "smithi112", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-04T21:53:54.098 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "inject_early_sigterm": "false", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "jaeger_agent_port": "6799", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "jaeger_tracing_enable": "false", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_aio": "true", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_align_min_size": "65536", 2024-09-04T21:53:54.099 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_block_align": "true", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_block_size": "4096", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_dio": "true", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_discard": "false", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_force_aio": "false", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_ignore_corruption": "false", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_max_write_bytes": "10485760", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_max_write_entries": "100", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_replay_from": "0", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-04T21:53:54.100 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_write_header_frequency": "0", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journal_zero_on_create": "false", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journaler_prefetch_periods": "10", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journaler_prezero_periods": "5", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "journaler_write_head_interval": "15", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "", 2024-09-04T21:53:54.101 INFO:teuthology.orchestra.run.smithi112.stdout: "keyfile": "", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_backend": "rocksdb", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_default_stripe_size": "65536", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_fsck_on_mount": "false", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_max_bytes": "67108864", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_max_ops": "512", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_nid_prealloc": "1024", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_onode_map_size": "1024", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-04T21:53:54.102 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_sync_submit_transaction": "false", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "kstore_sync_transaction": "false", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "librados_thread_count": "2", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "lockdep": "false", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "lockdep_force_backtrace": "false", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_coarse_timestamps": "true", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_flush_on_exit": "false", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_graylog_host": "127.0.0.1", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_graylog_port": "12201", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_max_new": "1000", 2024-09-04T21:53:54.103 INFO:teuthology.orchestra.run.smithi112.stdout: "log_max_recent": "10000", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_stderr_prefix": "", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_stop_at_utilization": "0.970000", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_to_file": "true", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_to_graylog": "false", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_to_journald": "false", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_to_stderr": "false", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_to_syslog": "false", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "max_rotating_auth_attempts": "10", 2024-09-04T21:53:54.104 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_action_on_write_error": "1", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_alternate_name_max": "8192", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_asio_thread_count": "2", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_export_pin": "true", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_fragment_dirs": "true", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_fragment_interval": "5", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_interval": "10", 2024-09-04T21:53:54.105 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_max": "-1", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_max_until": "-1", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_merge_size": "50", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_midchunk": "0.300000", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_min_start": "0.200000", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_minchunk": "0.001000", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_mode": "0", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_need_max": "1.200000", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_need_min": "0.800000", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_overload_epochs": "2", 2024-09-04T21:53:54.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_split_bits": "3", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_split_size": "10000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_target_decay": "10.000000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-04T21:53:54.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_beacon_grace": "15.000000", 2024-09-04T21:53:54.108 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_beacon_interval": "4.000000", 2024-09-04T21:53:54.108 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_mid": "0.700000", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_quiesce_delay": "0", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_release_free_interval": "10", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_reservation": "0.050000", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-04T21:53:54.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_trim_interval": "1", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache_trim_threshold": "262144", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_client_prealloc_inos": "1000", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_connect_bootstrapping": "false", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_damage_table_max_entries": "10000", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_debug_auth_pins": "false", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_debug_frag": "false", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_debug_scatterstat": "false", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_debug_subtrees": "false", 2024-09-04T21:53:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_decay_halflife": "5.000000", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_default_dir_hash": "2", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_defer_session_stale": "true", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_deny_all_reconnect": "false", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dir_keys_per_op": "16384", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dir_max_commit_size": "10", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dir_max_entries": "0", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dir_prefetch": "true", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dump_cache_on_map": "false", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-04T21:53:54.111 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_early_reply": "true", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_enable_op_tracker": "true", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_enforce_unique_name": "true", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_export_ephemeral_random": "true", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-04T21:53:54.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_health_summarize_threshold": "10", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_inject_health_dummy": "false", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_inject_migrator_session_race": "false", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_join_fs": "", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_journal_format": "1", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_create_at": "0", 2024-09-04T21:53:54.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_dirfrag_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_export_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_import_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_journal_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_journal_expire_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_journal_replay_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_link_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_mdstable_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_openc_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_rename_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_kill_shutdown_at": "0", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_event_large_threshold": "524288", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_events_per_segment": "1024", 2024-09-04T21:53:54.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_max_events": "-1", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_max_segments": "128", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_pause": "false", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_segment_size": "0", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_trim_threshold": "128", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log_warn_factor": "2.000000", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_caps_per_client": "1048576", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_completed_flushes": "100000", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_completed_requests": "100000", 2024-09-04T21:53:54.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_export_size": "20971520", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_file_recover": "32", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_purge_files": "64", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_purge_ops": "8192", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_max_snaps_per_dir": "100", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_metrics_update_interval": "2", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_min_caps_per_client": "100", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_min_caps_working_set": "10000", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_numa_node": "-1", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_op_complaint_time": "30.000000", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_op_history_duration": "600", 2024-09-04T21:53:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_op_history_size": "20", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_op_history_slow_op_size": "20", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_op_log_threshold": "5", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_ping_grace": "15", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_ping_interval": "5", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_recall_max_caps": "30000", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_recall_warning_threshold": "262144", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_replay_interval": "1.000000", 2024-09-04T21:53:54.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_root_ino_gid": "0", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_root_ino_uid": "0", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-04T21:53:54.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_shutdown_check": "0", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_skip_ino": "0", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_snap_max_uid": "4294967294", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_snap_min_uid": "0", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_snap_rstat": "false", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_standby_replay_damaged": "false", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_symlink_recovery": "true", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-04T21:53:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_thrash_exports": "0", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_thrash_fragments": "0", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_tick_interval": "5.000000", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_valgrind_exit": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_verify_backtrace": "1", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_verify_scatter": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_wipe_sessions": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool_debug": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "memstore_device_bytes": "1073741824", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "memstore_page_set": "false", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "memstore_page_size": "65536", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_client_bytes": "134217728", 2024-09-04T21:53:54.120 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_client_messages": "512", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_disabled_modules": "", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_enable_op_tracker": "true", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_max_pg_creating": "1024", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_max_pg_num_change": "128", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_mds_bytes": "134217728", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_mds_messages": "128", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_mon_bytes": "134217728", 2024-09-04T21:53:54.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_mon_messages": "128", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_op_history_duration": "600", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_op_history_size": "20", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_op_log_threshold": "5", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_osd_bytes": "536870912", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_osd_messages": "8192", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_pool": "true", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_standby_modules": "true", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_stats_period": "5", 2024-09-04T21:53:54.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_stats_threshold": "5", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_tick_period": "2", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_allow_pool_delete": "true", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_allow_pool_size_one": "true", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_auth_validate_all_caps": "true", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-04T21:53:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_bytes": "104857600", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_directed_command_retry": "5", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_hunt_parallel": "3", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_log_interval": "1.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_ping_interval": "10.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_client_target_rank": "-1", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-04T21:53:54.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_level": "debug", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_file": "true", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_journald": "false", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_compact_on_bootstrap": "false", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_compact_on_start": "false", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_compact_on_trim": "true", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-04T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_cpu_threads": "4", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_crush_min_required_version": "hammer", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_daemon_bytes": "419430400", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_data_avail_crit": "5", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_data_avail_warn": "5", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_data_size_warn": "16106127360", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_dump_json": "false", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_dump_transactions": "false", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_extra_checks": "false", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-04T21:53:54.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_no_require_reef": "false", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_no_require_squid": "false", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_down_mkfs_grace": "120", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_down_uptime_grace": "60", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_election_default_strategy": "1", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_election_timeout": "5.000000", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_elector_ping_divisor": "2", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_enable_op_tracker": "true", 2024-09-04T21:53:54.127 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_fake_pool_delete": "false", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_force_quorum_join": "false", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_globalid_prealloc": "10000", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_health_detail_to_clog": "false", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_health_log_update_period": "5", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_health_max_detail": "50", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_health_to_clog": "true", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_health_to_clog_interval": "600", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_host": "172.21.15.112,[v2:172.21.15.112:3301,v1:172.21.15.112:6790],172.21.15.157", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_host_override": "", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_initial_members": "", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-04T21:53:54.128 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_lease": "5.000000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_log_full_interval": "50", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_log_max": "10000", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_log_max_summary": "50", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_log_epochs": "500", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-04T21:53:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_osd": "10000", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_pg_per_osd": "10000", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_pool_pg_num": "65536", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mds_force_trim_to": "0", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mds_skip_sanity": "false", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_memory_autotune": "true", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_memory_target": "2147483648", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mgr_beacon_grace": "30", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mgr_digest_period": "5", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mgr_inactive_grace": "60", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-04T21:53:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_min_osdmap_epochs": "500", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_op_complaint_time": "30", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_op_history_duration": "600", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_op_history_size": "20", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_op_history_slow_op_size": "20", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_op_log_threshold": "5", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_auto_mark_in": "false", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-04T21:53:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_cache_size": "500", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_down_out_interval": "600", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_force_trim_to": "0", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-04T21:53:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_min_down_reporters": "2", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_report_timeout": "900", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-04T21:53:54.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pg_stuck_threshold": "60", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-04T21:53:54.134 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_probe_timeout": "2.000000", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_reweight_max_change": "0.050000", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_reweight_max_osds": "4", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_scrub_interval": "86400", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_scrub_max_keys": "100", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_scrub_timeout": "300", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_session_timeout": "300", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_smart_report_timeout": "5", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_stat_smooth_intervals": "6", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-04T21:53:54.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_stretch_pool_min_size": "2", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_stretch_pool_size": "4", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_sync_debug": "false", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_sync_provider_kill_at": "0", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_sync_requester_kill_at": "0", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_sync_timeout": "60.000000", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_target_pg_per_osd": "100", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_tick_interval": "5", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_timecheck_interval": "300.000000", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-04T21:53:54.136 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_use_min_delay_socket": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_older_version_delay": "604800", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_misplaced": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_older_version": "true", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-04T21:53:54.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": "", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "motr_tracing_enabled": "false", 2024-09-04T21:53:54.138 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_op_threads": "3", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_cm": "false", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_device_name": "", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_dscp": "96", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_local_gid": "", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_port_num": "1", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_sl": "3", 2024-09-04T21:53:54.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_support_srq": "true", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_rdma_type": "ib", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_async_reap_threshold": "5", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_before_connect": "false", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_ipv4": "true", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_ipv6": "false", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_msgr1": "true", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_msgr2": "true", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_port_max": "7568", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_port_min": "6800", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_retry_count": "3", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_bind_retry_delay": "5", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_blackhole_client": "false", 2024-09-04T21:53:54.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_blackhole_mds": "false", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_blackhole_mgr": "false", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_blackhole_mon": "false", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_blackhole_osd": "false", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_client_mode": "crc secure", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_cluster_mode": "crc secure", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_cluster_type": "", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_compress_secure": "false", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_connection_idle_timeout": "900", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_connection_ready_timeout": "10", 2024-09-04T21:53:54.141 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_crc_data": "true", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_crc_header": "true", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_die_on_bad_msg": "false", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_die_on_bug": "true", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_die_on_old_message": "true", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_die_on_skipped_message": "false", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_coremask": "0xF", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_enable_tso": "true", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-04T21:53:54.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_hugepages": "", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_lro": "true", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_memory_channel": "4", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_pmd": "", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_port_id": "0", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_dump_on_send": "false", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_initial_backoff": "0.200000", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_inject_delay_max": "1.000000", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-04T21:53:54.143 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_inject_delay_type": "", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_inject_network_congestion": "0", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_inject_socket_failures": "5000", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_learn_addr_from_peer": "true", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_max_accept_failures": "4", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_max_backoff": "15.000000", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_mon_client_mode": "secure crc", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_mon_service_mode": "secure crc", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_osd_compress_min_size": "1024", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_osd_compress_mode": "none", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_pq_min_cost": "65536", 2024-09-04T21:53:54.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_public_type": "", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_service_mode": "crc secure", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_tcp_listen_backlog": "512", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_tcp_nodelay": "true", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_tcp_rcvbuf": "0", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ms_type": "async+posix", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "no_config_file": "false", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_completion_locks_per_session": "32", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_inflight_ops": "1024", 2024-09-04T21:53:54.145 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_tick_interval": "5.000000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter_timeout": "10.000000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "objectstore_blackhole": "false", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "openssl_engine_opts": "", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_delay_time": "5.000000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_hist_halflife": "1000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_max_low_ops": "2", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_max_ops": "4", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_agent_slop": "0.020000", 2024-09-04T21:53:54.146 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_asio_thread_count": "2", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_async_recovery_min_cost": "100", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_backfill_scan_max": "512", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_backfill_scan_min": "64", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_backoff_on_degraded": "false", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_backoff_on_peering": "false", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_backoff_on_unfound": "true", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_beacon_report_interval": "300", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bench_duration": "30", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bench_max_block_size": "67108864", 2024-09-04T21:53:54.147 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_blkin_trace_all": "false", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_check_for_log_corruption": "false", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_class_update_on_start": "true", 2024-09-04T21:53:54.148 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_client_message_cap": "256", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_client_message_size_cap": "524288000", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_client_op_priority": "63", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_client_watch_timeout": "30", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_command_max_records": "256", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_compact_on_start": "false", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_crush_update_on_start": "true", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_crush_update_weight_set": "true", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-04T21:53:54.149 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_feed_pullee": "-1", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_misdirected_ops": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_no_acting_change": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_no_purge_strays": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_op_order": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-04T21:53:54.150 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_shutdown": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_trim_objects": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_verify_snaps": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-04T21:53:54.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_keys": "1024", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_stride": "524288", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_default_notify_timeout": "30", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_delete_sleep": "0.000000", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-04T21:53:54.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_discard_disconnected_ops": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_ec_partial_reads": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_enable_op_tracker": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_fast_info": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_fast_shutdown": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-04T21:53:54.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_function_tracing": "false", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_grace": "20", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_interval": "6", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_min_peers": "10", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_min_size": "2000", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_stale": "600", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_hit_set_max_size": "100000", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_hit_set_min_size": "1000", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-04T21:53:54.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_journal_size": "5120", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_kill_backfill_at": "0", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_map_cache_size": "50", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_map_dedup": "true", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_map_message_max": "40", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_map_share_max_epochs": "40", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_attr_name_len": "100", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_attr_size": "0", 2024-09-04T21:53:54.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_backfills": "1", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_markdown_count": "5", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_markdown_period": "600", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_object_name_len": "2048", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_object_namespace_len": "256", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_object_size": "134217728", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_pg_blocked_by": "16", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_pg_log_entries": "10000", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_pgls": "1024", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_push_cost": "8388608", 2024-09-04T21:53:54.156 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_push_objects": "10", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_scrubs": "3", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_trimming_pgs": "2", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_write_op_reply_len": "64", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_max_write_size": "90", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-04T21:53:54.157 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_profile": "balanced", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_base": "805306368", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_cache_min": "134217728", 2024-09-04T21:53:54.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_target": "4294967296", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_target_autotune": "false", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_min_pg_log_entries": "250", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_min_recovery_priority": "0", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mon_report_interval": "5", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_num_cache_shards": "32", 2024-09-04T21:53:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_num_op_tracker_shard": "32", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_numa_auto_affinity": "true", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_numa_node": "-1", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_numa_prefer_iface": "true", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_objecter_finishers": "1", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_objectstore": "bluestore", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_objectstore_fuse": "false", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_objectstore_tracing": "false", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_complaint_time": "30.000000", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_history_duration": "600", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_history_size": "20", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_history_slow_op_size": "20", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-04T21:53:54.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_log_threshold": "5", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_num_shards": "0", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_num_shards_hdd": "5", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_num_shards_ssd": "8", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_pq_min_cost": "65536", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_queue_cut_off": "high", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_op_thread_timeout": "15", 2024-09-04T21:53:54.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_open_classes_on_start": "true", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_os_flags": "0", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_peering_op_priority": "255", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_delete_cost": "1048576", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_delete_priority": "5", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_log_trim_max": "10000", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_log_trim_min": "100", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-04T21:53:54.162 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_crimson": "false", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-04T21:53:54.163 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_flags": "0", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_min_size": "0", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_pg_num": "32", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_pgp_num": "0", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_read_ratio": "70", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_size": "2", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_default_type": "replicated", 2024-09-04T21:53:54.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_push_per_object_cost": "1000", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recover_clone_overlap": "true", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_cost": "20971520", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_max_active": "0", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_max_single_start": "1", 2024-09-04T21:53:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_op_priority": "3", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_priority": "5", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_sleep": "0.000000", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_repair_during_recovery": "false", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_requested_scrub_priority": "120", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_auto_repair": "false", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-04T21:53:54.166 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_begin_hour": "0", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_begin_week_day": "0", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_chunk_max": "25", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_chunk_min": "5", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_cost": "52428800", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_during_recovery": "false", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_end_hour": "0", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_end_week_day": "0", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_event_cost": "4096", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_invalid_stats": "true", 2024-09-04T21:53:54.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_max_preemptions": "5", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_priority": "5", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_scrub_sleep": "0.000000", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_skip_data_digest": "false", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_smart_report_timeout": "5", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_snap_trim_cost": "1048576", 2024-09-04T21:53:54.168 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_snap_trim_priority": "5", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_target_transaction_size": "30", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-04T21:53:54.169 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_tracing": "false", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_use_stale_snap": "false", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "osdc_blkin_trace_all": "false", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_kill_at": "0", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_max_join_drift": "10", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_min": "500", 2024-09-04T21:53:54.170 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_min_wait": "0.050000", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_propose_interval": "1.000000", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_service_trim_max": "500", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_service_trim_min": "250", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_stash_full_interval": "25", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_trim_max": "500", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "paxos_trim_min": "250", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "perf": "true", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-04T21:53:54.171 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "-", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrv": "", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "public_bind_addr": "-", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "public_network": "", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "public_network_interface": "", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "qat_compressor_busy_polling": "false", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "qat_compressor_enabled": "false", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "qat_compressor_session_max_number": "256", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "rados_mon_op_timeout": "0", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "rados_osd_op_timeout": "0", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "rados_tracing": "false", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_atime_update_interval": "60", 2024-09-04T21:53:54.172 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_balance_parent_reads": "false", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_balance_snap_reads": "false", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_blkin_trace_all": "false", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache": "true", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-04T21:53:54.173 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_policy": "writearound", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_size": "33554432", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_clone_copy_on_read": "false", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_compression_hint": "none", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_concurrent_management_ops": "10", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_clone_format": "auto", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_data_pool": "", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_features": "61", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_format": "2", 2024-09-04T21:53:54.174 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_map_options": "", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_order": "22", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_pool": "rbd", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_stripe_count": "0", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_default_stripe_unit": "0", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_enable_alloc_hint": "true", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_io_scheduler": "simple", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-04T21:53:54.175 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_order": "24", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_pool": "", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_journal_splay_width": "4", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_localize_parent_reads": "false", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_localize_snap_reads": "false", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-04T21:53:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-04T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-04T21:53:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_mtime_update_interval": "60", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_non_blocking_aio": "true", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_op_thread_timeout": "60", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_op_threads": "1", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_parent_cache_enabled": "false", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_plugins": "", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_bps_burst": "0", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_bps_limit": "0", 2024-09-04T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_exclude_ops": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_iops_burst": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_iops_limit": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-04T21:53:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_read_from_replica_policy": "default", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_skip_partial_discard": "true", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_tracing": "false", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_validate_names": "true", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd_validate_pool": "true", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "restapi_base_url": "", 2024-09-04T21:53:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "restapi_log_level": "", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_acl_grants_max_num": "100", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_admin_entry": "admin", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_asio_assert_yielding": "false", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_backend_store": "rados", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_barbican_url": "", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_beast_enable_async": "true", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_counters_cache": "false", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-04T21:53:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_cache_enabled": "true", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_cache_expiry_interval": "900", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_cache_lru_size": "25000", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_config_store": "rados", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_content_length_compat": "false", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_copy_obj_progress": "true", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-04T21:53:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_cors_rules_max_num": "100", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_cross_domain_policy": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_addr": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_password": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_kmip_username": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_require_ssl": "true", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-04T21:53:54.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_addr": "", 2024-09-04T21:53:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_auth": "token", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_namespace": "", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_prefix": "", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_token_file": "", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_curl_buffersize": "524288", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_curl_low_speed_time": "300", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-04T21:53:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-04T21:53:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_log_changes_size": "1000", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_log_num_shards": "128", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_log_window": "30", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-04T21:53:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_disable_s3select": "false", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-04T21:53:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dns_name": "", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dns_s3website_name": "", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dynamic_resharding": "true", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_gc_threads": "true", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_lc_threads": "true", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_mdsearch": "true", 2024-09-04T21:53:54.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_ops_log": "false", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_quota_threads": "true", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_static_website": "false", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enable_usage_log": "false", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_enforce_swift_acls": "true", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_exit_timeout_secs": "120", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_expose_bucket": "false", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_extended_http_attrs": "", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_filter": "none", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_frontends": "beast port=7480", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_max_deferred": "50", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-04T21:53:54.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_max_objs": "32", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_gc_processor_period": "3600", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_init_timeout": "300", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-04T21:53:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_kafka_connection_idle": "300", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_domain": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_password": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_password_path": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_project": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_tenant": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_token": "", 2024-09-04T21:53:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_token_path": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_admin_user": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_barbican_domain": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_barbican_password": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_barbican_project": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_barbican_user": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_url": "", 2024-09-04T21:53:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_debug_interval": "-1", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_lock_max_time": "90", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_max_objs": "32", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_max_rules": "1000", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_max_worker": "3", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lc_thread_delay": "0", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ldap_dnattr": "uid", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ldap_searchfilter": "", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ldap_uri": "", 2024-09-04T21:53:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_log_http_headers": "", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_log_object_name_utc": "false", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_attr_name_len": "0", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_attr_size": "0", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_chunk_size": "4194304", 2024-09-04T21:53:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_listing_results": "1000", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_notify_retries": "10", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_put_param_size": "1048576", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_put_size": "5368709120", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_max_slo_entries": "1000", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_md_log_max_shards": "64", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-04T21:53:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_mp_lock_max_time": "600", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_max_gc": "300", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-04T21:53:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_num_async_rados_threads": "32", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_num_control_oids": "8", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_numa_node": "-1", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_objexp_chunk_size": "100", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_objexp_gc_interval": "600", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-04T21:53:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_op_thread_timeout": "600", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_op_tracing": "false", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_opa_token": "", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_opa_url": "", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_opa_verify_ssl": "true", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ops_log_rados": "false", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_ops_log_socket_path": "", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_period_push_interval": "2.000000", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-04T21:53:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_posix_cache_lanes": "3", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_posix_cache_partitions": "3", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_print_continue": "true", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-04T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_rados_tracing": "false", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_realm": "", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_realm_id": "", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_region": "", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_request_uri": "", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_reshard_batch_size": "64", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-04T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_reshard_max_aio": "128", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_reshard_num_logs": "16", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_reshard_thread_interval": "600", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_resolve_cname": "false", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_run_sync_thread": "true", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-04T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_scheduler_type": "throttler", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_script_uri": "", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_service_provider_name": "", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_client_id": "", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_client_secret": "", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_entry": "sts", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_key": "sts", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-04T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_min_session_duration": "900", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sts_token_introspection_url": "", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_account_in_url": "false", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_auth_entry": "auth", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_auth_url": "", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_custom_header": "", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_need_stats": "true", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_tenant_name": "", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_token_expiration": "86400", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_url": "", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_url_prefix": "swift", 2024-09-04T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_lease_period": "120", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-04T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_thread_pool_size": "512", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_comment": "", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_createby": "", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_encoding": "", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_flag": "false", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_origin": "", 2024-09-04T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_torrent_tracker": "", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_trust_forwarded_https": "false", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_usage_max_shards": "32", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_usage_max_user_shards": "1", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_use_opa_authz": "false", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_counters_cache": "false", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-04T21:53:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_max_buckets": "1000", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_policies_max_num": "100", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_user_unique_email": "true", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_verify_ssl": "true", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-04T21:53:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_zone": "", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_zone_id": "", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_zonegroup": "", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_zonegroup_id": "", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgwlc_auto_session_clear": "true", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_block_size": "4096", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-04T21:53:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cache_size": "536870912", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_index_type": "binary_search", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-04T21:53:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_partition_filters": "false", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_perf": "false", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "run_dir": "/var/run/ceph", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_block_create": "true", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_cache_lru_size": "67108864", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_default_max_object_size": "16777216", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-04T21:53:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_device_size": "53687091200", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_full_integrity_check": "false", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_journal_batch_capacity": "16", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_main_device_type": "SSD", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-04T21:53:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-04T21:53:54.216 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "seastore_segment_size": "67108864", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "set_keepcaps": "false", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "setgroup": "", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "setuser": "", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "setuser_match_path": "", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "thp": "false", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "threadpool_default_timeout": "60", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "throttler_perf_counter": "true", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "tmp_dir": "/tmp", 2024-09-04T21:53:54.217 INFO:teuthology.orchestra.run.smithi112.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-04T21:53:54.218 INFO:teuthology.orchestra.run.smithi112.stdout: "uadk_compressor_enabled": "false", 2024-09-04T21:53:54.218 INFO:teuthology.orchestra.run.smithi112.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-04T21:53:54.218 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:53:54.220 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-04T21:53:54.353 INFO:teuthology.orchestra.run.smithi112.stdout:mon.a: {} 2024-09-04T21:53:54.354 INFO:teuthology.orchestra.run.smithi112.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-04T21:53:54.367 INFO:teuthology.orchestra.run.smithi112.stdout:mon.b: {} 2024-09-04T21:53:54.367 INFO:teuthology.orchestra.run.smithi112.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-04T21:53:54.375 INFO:teuthology.orchestra.run.smithi112.stdout:mon.c: {} 2024-09-04T21:53:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-04T21:53:54.390 DEBUG:teuthology.orchestra.run.smithi157:> type ceph-objectstore-tool 2024-09-04T21:53:54.397 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-04T21:53:54.397 DEBUG:teuthology.orchestra.run.smithi112:> type ceph-objectstore-tool 2024-09-04T21:53:54.402 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-04T21:53:54.403 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-09-04T21:53:54.414 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-09-04T21:53:54.414 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-04T21:53:54.414 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:53:54.414 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:53:54.414 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:53:54.415 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-04T21:53:54.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:53:54.415 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-04T21:53:54.416 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-04T21:53:54.416 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:53:54.417 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-04T21:53:54.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:53:54.421 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-04T21:53:54.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:53:54.422 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-09-04T21:53:54.443 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:54.439+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:53:54.459 INFO:teuthology.orchestra.run.smithi112.stdout:mapped writes DISABLED 2024-09-04T21:53:54.460 INFO:teuthology.orchestra.run.smithi112.stdout:mapped reads DISABLED 2024-09-04T21:53:54.460 INFO:teuthology.orchestra.run.smithi112.stdout:Seed set to 1609 2024-09-04T21:53:54.517 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:54.515+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:54.560 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:53:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:53:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:53:54.561 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:53:54.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:53:54.577 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:53:54.577 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:53:54.594 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:53:54.594 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:53:54.611 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:53:54.612 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:53:54.618 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:54.616+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:54.628 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:53:54.628 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:53:54.648 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:53:54.648 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:53:54.670 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:53:54.670 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:53:54.681 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:54.681 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:53:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:53:54.694 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:53:54.719 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:54.715+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:54.820 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:54.820+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:54.843 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:54.843 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:54.855 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:53:54.943 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:54.944+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:55.009 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.010 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:53:55.023 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:55.020+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:55.036 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:53:55.123 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:55.120+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:55.151 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.151 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.163 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:53:55.252 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:55.252+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:53:55.318 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.319 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.325 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:55.324+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:53:55.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:53:55.425 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:55.424+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:55.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-04T21:53:55.498 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.498 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.512 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:53:55.527 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:55.523+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:55.627 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:55.623+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:55.669 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.670 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.682 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:53:55.731 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:55.732+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:55.802 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.802 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:53:55.830 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:55.827+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:53:55.931 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:55.928+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:53:55.984 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:55.984 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:55.998 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:53:56.039 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:56.040+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:56.133 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:56.131+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:56.160 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:56.160 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:56.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:53:56.234 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:56.231+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:56.295 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:56.295 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:56.309 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:53:56.358 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:56.355+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:53:56.436 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:56.436+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:56.441 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:56.441 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:56.456 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:53:56.542 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:56.539+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:56.620 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:56.621 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:56.635 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:53:56.637 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:56.636+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:56.738 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:56.735+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:56.807 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:56.807 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:56.821 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:53:56.866 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:56.863+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:56.940 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:56.940+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:53:56.989 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:56.989 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.004 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:53:57.040 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:57.040+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:57.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:53:57.128 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:57.129 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.141 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:57.140+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:57.143 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:53:57.243 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:57.240+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:57.269 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:57.269 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.284 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:53:57.344 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:57.339+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:57.406 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:57.406 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.421 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:53:57.444 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:57.444+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:57.545 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:57.544+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:57.586 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:57.587 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.601 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:53:57.650 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:57.647+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:57.724 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:57.725 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.738 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:53:57.783 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:57.784+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:57.848 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:57.848+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:53:57.850 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:57.850 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:57.862 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:53:57.950 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:57.947+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:53:58.027 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.027 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:53:58.083 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:58.084+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:58.149 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.149 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.152 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:58.147+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:58.161 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:53:58.252 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:58.247+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:58.318 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.318 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:53:58.337 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:53:58.353 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:58.352+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:58.446 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.446 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.454 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:58.451+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:58.460 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:53:58.555 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:58.551+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:58.574 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.574 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.588 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:53:58.655 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:53:58.656+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:53:58.710 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.710 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:53:58.757 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:53:58.751+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:53:58.857 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:53:58.856+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:53:58.891 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:58.892 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:58.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:53:58.959 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:58.955+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:53:59.060 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:59.059+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:59.060 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:59.060 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:59.075 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:53:59.161 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:59.160+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:59.197 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:59.197 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:59.213 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:53:59.241 INFO:teuthology.orchestra.run.smithi112.stdout:1 writesame 0x2195f99 thru 0x219d5a3 data_size 0x7(0x760b bytes) 2024-09-04T21:53:59.254 INFO:teuthology.orchestra.run.smithi112.stdout:2 write 0x629e1a6 thru 0x62a9008 (0xae63 bytes) 2024-09-04T21:53:59.260 INFO:teuthology.orchestra.run.smithi112.stdout:4 punch from 0x4c2739a to 0x4c33053, (0xbcb9 bytes) 2024-09-04T21:53:59.260 INFO:teuthology.orchestra.run.smithi112.stdout:5 read 0x1fc9188 thru 0x1fd3d43 (0xabbc bytes) 2024-09-04T21:53:59.261 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:59.260+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:59.270 INFO:teuthology.orchestra.run.smithi112.stdout:7 write 0x80db7f8 thru 0x80df118 (0x3921 bytes) 2024-09-04T21:53:59.363 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:59.359+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:59.366 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:59.366 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:59.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:53:59.467 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:59.468+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:59.479 INFO:teuthology.orchestra.run.smithi112.stdout:8 writesame 0x62fc3d8 thru 0x6305cab data_size 0x1(0x98d4 bytes) 2024-09-04T21:53:59.511 INFO:teuthology.orchestra.run.smithi112.stdout:9 read 0x402898d thru 0x402cb30 (0x41a4 bytes) 2024-09-04T21:53:59.536 INFO:teuthology.orchestra.run.smithi112.stdout:10 write 0xdfa337e thru 0xdfa42e9 (0xf6c bytes) 2024-09-04T21:53:59.540 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:59.540 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:59.542 INFO:teuthology.orchestra.run.smithi112.stdout:14 writesame 0x4966eb6 thru 0x496d9ac data_size 0x1(0x6af7 bytes) 2024-09-04T21:53:59.554 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:53:59.560 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:53:59.564 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:59.564+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:53:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:53:59.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:53:59.571 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:53:59.572 INFO:teuthology.orchestra.run.smithi112.stdout:16 writesame 0xdf996 thru 0xe8a2b data_size 0x3e(0x9096 bytes) 2024-09-04T21:53:59.580 INFO:teuthology.orchestra.run.smithi112.stdout:truncating to largest ever: 0x8cabf4f 2024-09-04T21:53:59.587 INFO:teuthology.orchestra.run.smithi112.stdout:17 trunc from 0xdfa42ea to 0x8cabf4f 2024-09-04T21:53:59.599 INFO:teuthology.orchestra.run.smithi112.stdout:18 writesame 0x601914 thru 0x60c351 data_size 0x4d(0xaa3e bytes) 2024-09-04T21:53:59.617 INFO:teuthology.orchestra.run.smithi112.stdout:21 write 0xecebe01 thru 0xecf2ca4 (0x6ea4 bytes) 2024-09-04T21:53:59.621 INFO:teuthology.orchestra.run.smithi112.stdout:22 punch from 0x87c97fd to 0x87d5936, (0xc139 bytes) 2024-09-04T21:53:59.621 INFO:teuthology.orchestra.run.smithi112.stdout:24 punch from 0xbf080b9 to 0xbf0c5dc, (0x4523 bytes) 2024-09-04T21:53:59.621 INFO:teuthology.orchestra.run.smithi112.stdout:25 write 0x3404f5 thru 0x34b41e (0xaf2a bytes) 2024-09-04T21:53:59.623 INFO:teuthology.orchestra.run.smithi112.stdout:26 writesame 0x81ba1dd thru 0x81c3b29 data_size 0x2f(0x994d bytes) 2024-09-04T21:53:59.626 INFO:teuthology.orchestra.run.smithi112.stdout:27 read 0x6283155 thru 0x6291350 (0xe1fc bytes) 2024-09-04T21:53:59.627 INFO:teuthology.orchestra.run.smithi112.stdout:30 read 0x797e278 thru 0x79899f1 (0xb77a bytes) 2024-09-04T21:53:59.642 INFO:teuthology.orchestra.run.smithi112.stdout:31 trunc from 0xecf2ca5 to 0x37558cd 2024-09-04T21:53:59.661 INFO:teuthology.orchestra.run.smithi112.stdout:32 punch from 0x16ff40d to 0x1709fdd, (0xabd0 bytes) 2024-09-04T21:53:59.661 INFO:teuthology.orchestra.run.smithi112.stdout:33 write 0x8631c9 thru 0x8712ee (0xe126 bytes) 2024-09-04T21:53:59.665 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:53:59.664+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:53:59.682 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:59.682 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:59.696 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:53:59.704 INFO:teuthology.orchestra.run.smithi112.stdout:34 write 0xe7dc6f9 thru 0xe7e160c (0x4f14 bytes) 2024-09-04T21:53:59.711 INFO:teuthology.orchestra.run.smithi112.stdout:37 read 0x28b2778 thru 0x28c1f29 (0xf7b2 bytes) 2024-09-04T21:53:59.711 INFO:teuthology.orchestra.run.smithi112.stdout:39 read 0xa88bd5f thru 0xa899775 (0xda17 bytes) 2024-09-04T21:53:59.711 INFO:teuthology.orchestra.run.smithi112.stdout:40 write 0x9d3dc2a thru 0x9d4aee8 (0xd2bf bytes) 2024-09-04T21:53:59.746 INFO:teuthology.orchestra.run.smithi112.stdout:41 trunc from 0xe7e160d to 0x36266d6 2024-09-04T21:53:59.766 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:53:59.763+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:53:59.768 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:53:59.776 INFO:teuthology.orchestra.run.smithi112.stdout:44 trunc from 0x36266d6 to 0xd7ff84 2024-09-04T21:53:59.820 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:53:59.820 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:53:59.829 INFO:teuthology.orchestra.run.smithi112.stdout:45 write 0xbc474a7 thru 0xbc531f8 (0xbd52 bytes) 2024-09-04T21:53:59.834 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:53:59.837 INFO:teuthology.orchestra.run.smithi112.stdout:truncating to largest ever: 0xc6fc241 2024-09-04T21:53:59.838 INFO:teuthology.orchestra.run.smithi112.stdout:46 trunc from 0xbc531f9 to 0xc6fc241 2024-09-04T21:53:59.851 INFO:teuthology.orchestra.run.smithi112.stdout:47 write 0x58db234 thru 0x58df738 (0x4505 bytes) 2024-09-04T21:53:59.857 INFO:teuthology.orchestra.run.smithi112.stdout:48 write 0x266bd6e thru 0x2677900 (0xbb93 bytes) 2024-09-04T21:53:59.864 INFO:teuthology.orchestra.run.smithi112.stdout:49 write 0x7969028 thru 0x7975ad7 (0xcab0 bytes) 2024-09-04T21:53:59.873 INFO:teuthology.orchestra.run.smithi112.stdout:51 read 0x2f3175b thru 0x2f3f881 (0xe127 bytes) 2024-09-04T21:53:59.873 INFO:teuthology.orchestra.run.smithi112.stdout:54 write 0x1140b73 thru 0x114acc4 (0xa152 bytes) 2024-09-04T21:53:59.878 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:53:59.875+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:53:59.896 INFO:teuthology.orchestra.run.smithi112.stdout:56 writesame 0xdb2ddd4 thru 0xdb33717 data_size 0x3a(0x5944 bytes) 2024-09-04T21:53:59.896 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:53:59.896 INFO:teuthology.orchestra.run.smithi112.stdout:58 punch from 0xbefc2f to 0xbff0e8, (0xf4b9 bytes) 2024-09-04T21:53:59.897 INFO:teuthology.orchestra.run.smithi112.stdout:60 punch from 0x122c370 to 0x1233fbc, (0x7c4c bytes) 2024-09-04T21:53:59.897 INFO:teuthology.orchestra.run.smithi112.stdout:truncating to largest ever: 0xdd2b2c4 2024-09-04T21:53:59.897 INFO:teuthology.orchestra.run.smithi112.stdout:61 trunc from 0xdb33718 to 0xdd2b2c4 2024-09-04T21:53:59.908 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:53:59.908 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:53:59.909 INFO:teuthology.orchestra.run.smithi112.stdout:64 read 0xcd2f343 thru 0xcd35921 (0x65df bytes) 2024-09-04T21:53:59.909 INFO:teuthology.orchestra.run.smithi112.stdout:65 read 0x6e18c20 thru 0x6e27a07 (0xede8 bytes) 2024-09-04T21:53:59.944 INFO:teuthology.orchestra.run.smithi112.stdout:66 trunc from 0xdd2b2c4 to 0x3eaa1f 2024-09-04T21:53:59.968 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:53:59.968+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:00.001 INFO:teuthology.orchestra.run.smithi112.stdout:67 writesame 0x80177c7 thru 0x801a247 data_size 0x27(0x2a81 bytes) 2024-09-04T21:54:00.001 INFO:teuthology.orchestra.run.smithi112.stdout:69 read 0x31d072d thru 0x31d0af8 (0x3cc bytes) 2024-09-04T21:54:00.004 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.004 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.018 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:00.026 INFO:teuthology.orchestra.run.smithi112.stdout:70 write 0xc65eab6 thru 0xc65ec7b (0x1c6 bytes) 2024-09-04T21:54:00.032 INFO:teuthology.orchestra.run.smithi112.stdout:72 read 0x12cd0e0 thru 0x12d3eae (0x6dcf bytes) 2024-09-04T21:54:00.032 INFO:teuthology.orchestra.run.smithi112.stdout:73 read 0x4bd8683 thru 0x4be6b84 (0xe502 bytes) 2024-09-04T21:54:00.033 INFO:teuthology.orchestra.run.smithi112.stdout:75 read 0x96f9ad3 thru 0x96f9fa0 (0x4ce bytes) 2024-09-04T21:54:00.033 INFO:teuthology.orchestra.run.smithi112.stdout:76 writesame 0x8a5e09b thru 0x8a5e91b data_size 0x1(0x881 bytes) 2024-09-04T21:54:00.041 INFO:teuthology.orchestra.run.smithi112.stdout:78 write 0x7f4e950 thru 0x7f5a039 (0xb6ea bytes) 2024-09-04T21:54:00.049 INFO:teuthology.orchestra.run.smithi112.stdout:79 read 0x51f717c thru 0x520667d (0xf502 bytes) 2024-09-04T21:54:00.049 INFO:teuthology.orchestra.run.smithi112.stdout:80 read 0x3a144df thru 0x3a1dad0 (0x95f2 bytes) 2024-09-04T21:54:00.049 INFO:teuthology.orchestra.run.smithi112.stdout:81 writesame 0x56d45dd thru 0x56d586c data_size 0xc(0x1290 bytes) 2024-09-04T21:54:00.068 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:00.063+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:00.072 INFO:teuthology.orchestra.run.smithi112.stdout:84 write 0xe2ac571 thru 0xe2b307c (0x6b0c bytes) 2024-09-04T21:54:00.079 INFO:teuthology.orchestra.run.smithi112.stdout:85 punch from 0xad90bbd to 0xad9653f, (0x5982 bytes) 2024-09-04T21:54:00.080 INFO:teuthology.orchestra.run.smithi112.stdout:87 write 0xb29e29e thru 0xb2a5131 (0x6e94 bytes) 2024-09-04T21:54:00.086 INFO:teuthology.orchestra.run.smithi112.stdout:89 write 0x92374f4 thru 0x9241e95 (0xa9a2 bytes) 2024-09-04T21:54:00.119 INFO:teuthology.orchestra.run.smithi112.stdout:90 trunc from 0xe2b307d to 0x389f6e1 2024-09-04T21:54:00.142 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.142 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:00.177 INFO:teuthology.orchestra.run.smithi112.stdout:91 writesame 0xdcb7508 thru 0xdcb9367 data_size 0x18(0x1e60 bytes) 2024-09-04T21:54:00.177 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.177 INFO:teuthology.orchestra.run.smithi112.stdout:94 read 0xa2726a0 thru 0xa2776e6 (0x5047 bytes) 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:97 read 0x382c5b1 thru 0x383470e (0x815e bytes) 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:98 read 0x1820372 thru 0x1826cd2 (0x6961 bytes) 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:100 read 0x3dbb454 thru 0x3dbbbc8 (0x775 bytes) 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:101 read 0xc6c3b5 thru 0xc71102 (0x4d4e bytes) 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:102 punch from 0x25a61b1 to 0x25a9eec, (0x3d3b bytes) 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.178 INFO:teuthology.orchestra.run.smithi112.stdout:104 writesame 0xd9bd6d8 thru 0xd9cb4a1 data_size 0x3(0xddca bytes) 2024-09-04T21:54:00.187 INFO:teuthology.orchestra.run.smithi112.stdout:105 write 0xce8ce67 thru 0xce996e1 (0xc87b bytes) 2024-09-04T21:54:00.194 INFO:teuthology.orchestra.run.smithi112.stdout:106 read 0x1685f75 thru 0x1695259 (0xf2e5 bytes) 2024-09-04T21:54:00.195 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.195 INFO:teuthology.orchestra.run.smithi112.stdout:108 write 0xa0388bc thru 0xa0452b1 (0xc9f6 bytes) 2024-09-04T21:54:00.198 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:00.195+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:00.200 INFO:teuthology.orchestra.run.smithi112.stdout:112 punch from 0x379e83d to 0x37ad462, (0xec25 bytes) 2024-09-04T21:54:00.201 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.201 INFO:teuthology.orchestra.run.smithi112.stdout:114 read 0x961519a thru 0x961f63d (0xa4a4 bytes) 2024-09-04T21:54:00.201 INFO:teuthology.orchestra.run.smithi112.stdout:115 read 0x8818f47 thru 0x88199e4 (0xa9e bytes) 2024-09-04T21:54:00.201 INFO:teuthology.orchestra.run.smithi112.stdout:116 punch from 0x7dea636 to 0x7df19b3, (0x737d bytes) 2024-09-04T21:54:00.201 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.217 INFO:teuthology.orchestra.run.smithi112.stdout:119 trunc from 0xdcb9368 to 0x64a5a92 2024-09-04T21:54:00.251 INFO:teuthology.orchestra.run.smithi112.stdout:121 trunc from 0x64a5a92 to 0xbe20905 2024-09-04T21:54:00.263 INFO:teuthology.orchestra.run.smithi112.stdout:123 write 0xa543a47 thru 0xa54738e (0x3948 bytes) 2024-09-04T21:54:00.270 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:00.267+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:00.270 INFO:teuthology.orchestra.run.smithi112.stdout:124 punch from 0x82afb3 to 0x832854, (0x78a1 bytes) 2024-09-04T21:54:00.271 INFO:teuthology.orchestra.run.smithi112.stdout:127 writesame 0x21363a4 thru 0x213a452 data_size 0x1(0x40af bytes) 2024-09-04T21:54:00.293 INFO:teuthology.orchestra.run.smithi112.stdout:128 write 0x52b14f7 thru 0x52bdee7 (0xc9f1 bytes) 2024-09-04T21:54:00.321 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.321 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.326 INFO:teuthology.orchestra.run.smithi112.stdout:132 write 0xeb81870 thru 0xeb820fc (0x88d bytes) 2024-09-04T21:54:00.335 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:00.336 INFO:teuthology.orchestra.run.smithi112.stdout:133 punch from 0xa329eea to 0xa32d552, (0x3668 bytes) 2024-09-04T21:54:00.336 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.336 INFO:teuthology.orchestra.run.smithi112.stdout:137 punch from 0x398e961 to 0x399717e, (0x881d bytes) 2024-09-04T21:54:00.362 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:54:00.367 INFO:teuthology.orchestra.run.smithi112.stdout:138 trunc from 0xeb820fd to 0x27c39b1 2024-09-04T21:54:00.371 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:00.368+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:00.390 INFO:teuthology.orchestra.run.smithi112.stdout:139 writesame 0xac74b7 thru 0xacdb32 data_size 0xe(0x667c bytes) 2024-09-04T21:54:00.396 INFO:teuthology.orchestra.run.smithi112.stdout:140 punch from 0xc69216 to 0xc6f0f4, (0x5ede bytes) 2024-09-04T21:54:00.397 INFO:teuthology.orchestra.run.smithi112.stdout:141 write 0xabf87d thru 0xac3d56 (0x44da bytes) 2024-09-04T21:54:00.398 INFO:teuthology.orchestra.run.smithi112.stdout:142 write 0x1700659 thru 0x1707569 (0x6f11 bytes) 2024-09-04T21:54:00.431 INFO:teuthology.orchestra.run.smithi112.stdout:144 trunc from 0x27c39b1 to 0xdabe374 2024-09-04T21:54:00.439 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.456 INFO:teuthology.orchestra.run.smithi112.stdout:146 trunc from 0xdabe374 to 0x29d2c08 2024-09-04T21:54:00.472 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:00.472+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:00.486 INFO:teuthology.orchestra.run.smithi112.stdout:147 write 0xa54d1e2 thru 0xa54e9c9 (0x17e8 bytes) 2024-09-04T21:54:00.491 INFO:teuthology.orchestra.run.smithi112.stdout:148 writesame 0x3267aae thru 0x3274bfa data_size 0x1(0xd14d bytes) 2024-09-04T21:54:00.501 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.501 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.516 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:00.517 INFO:tasks.ceph.osd.7:Stopped 2024-09-04T21:54:00.517 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-04T21:54:00.518 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:54:00.518 DEBUG:teuthology.orchestra.run.smithi157:> type ceph-objectstore-tool 2024-09-04T21:54:00.519 INFO:teuthology.orchestra.run.smithi112.stdout:149 write 0xabe26e3 thru 0xabf00f2 (0xda10 bytes) 2024-09-04T21:54:00.527 INFO:teuthology.orchestra.run.smithi112.stdout:150 trunc from 0xabf00f3 to 0xd53ff74 2024-09-04T21:54:00.535 INFO:teuthology.orchestra.run.smithi112.stdout:153 writesame 0xb855132 thru 0xb863ad9 data_size 0x4(0xe9a8 bytes) 2024-09-04T21:54:00.547 INFO:teuthology.orchestra.run.smithi112.stdout:154 write 0x6344859 thru 0x63546fd (0xfea5 bytes) 2024-09-04T21:54:00.568 INFO:teuthology.orchestra.run.smithi112.stdout:155 trunc from 0xd53ff74 to 0x215120c 2024-09-04T21:54:00.569 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-04T21:54:00.569 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-04T21:54:00.572 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:00.572+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:00.605 INFO:teuthology.orchestra.run.smithi112.stdout:156 writesame 0xbc72520 thru 0xbc72e73 data_size 0x6(0x954 bytes) 2024-09-04T21:54:00.608 INFO:teuthology.orchestra.run.smithi112.stdout:157 trunc from 0xbc72e74 to 0xda496ea 2024-09-04T21:54:00.615 INFO:teuthology.orchestra.run.smithi112.stdout:158 read 0x588df3b thru 0x589452a (0x65f0 bytes) 2024-09-04T21:54:00.633 INFO:teuthology.orchestra.run.smithi112.stdout:159 trunc from 0xda496ea to 0xc9cb01 2024-09-04T21:54:00.639 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.639 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.646 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.647 INFO:teuthology.orchestra.run.smithi112.stdout:161 read 0x6da0f thru 0x7482e (0x6e20 bytes) 2024-09-04T21:54:00.647 INFO:teuthology.orchestra.run.smithi112.stdout:162 read 0x6bb969 thru 0x6c4749 (0x8de1 bytes) 2024-09-04T21:54:00.647 INFO:teuthology.orchestra.run.smithi112.stdout:163 read 0x96c41c thru 0x977706 (0xb2eb bytes) 2024-09-04T21:54:00.655 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:00.671 INFO:teuthology.orchestra.run.smithi112.stdout:164 write 0xaabf312 thru 0xaac607b (0x6d6a bytes) 2024-09-04T21:54:00.674 INFO:teuthology.orchestra.run.smithi112.stdout:165 write 0x9d8f258 thru 0x9d9ac09 (0xb9b2 bytes) 2024-09-04T21:54:00.678 INFO:teuthology.orchestra.run.smithi112.stdout:166 write 0x3844b20 thru 0x3853472 (0xe953 bytes) 2024-09-04T21:54:00.683 INFO:teuthology.orchestra.run.smithi112.stdout:truncating to largest ever: 0xea96ccb 2024-09-04T21:54:00.690 INFO:teuthology.orchestra.run.smithi112.stdout:167 trunc from 0xaac607c to 0xea96ccb 2024-09-04T21:54:00.697 INFO:teuthology.orchestra.run.smithi112.stdout:168 read 0xa08ec90 thru 0xa08fb44 (0xeb5 bytes) 2024-09-04T21:54:00.697 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.698 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:00.695+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:00.716 INFO:teuthology.orchestra.run.smithi112.stdout:170 trunc from 0xea96ccb to 0x2eb0edd 2024-09-04T21:54:00.727 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.739 INFO:teuthology.orchestra.run.smithi112.stdout:172 writesame 0x632afe7 thru 0x632f7bf data_size 0x1(0x47d9 bytes) 2024-09-04T21:54:00.752 INFO:teuthology.orchestra.run.smithi112.stdout:173 writesame 0x96487bf thru 0x964b656 data_size 0x47(0x2e98 bytes) 2024-09-04T21:54:00.752 INFO:teuthology.orchestra.run.smithi112.stdout:174 read 0x2e2d7f6 thru 0x2e3d2d6 (0xfae1 bytes) 2024-09-04T21:54:00.752 INFO:teuthology.orchestra.run.smithi112.stdout:175 writesame 0xfc30da thru 0xfcbd4a data_size 0x1(0x8c71 bytes) 2024-09-04T21:54:00.774 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:00.772+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:00.777 INFO:teuthology.orchestra.run.smithi112.stdout:177 punch from 0x2d6f2a to 0x2e234c, (0xb422 bytes) 2024-09-04T21:54:00.777 INFO:teuthology.orchestra.run.smithi112.stdout:178 write 0x3a93ac8 thru 0x3aa071d (0xcc56 bytes) 2024-09-04T21:54:00.784 INFO:teuthology.orchestra.run.smithi112.stdout:179 write 0x51c851 thru 0x522925 (0x60d5 bytes) 2024-09-04T21:54:00.789 INFO:teuthology.orchestra.run.smithi112.stdout:180 write 0x13264bf thru 0x13323fd (0xbf3f bytes) 2024-09-04T21:54:00.794 INFO:teuthology.orchestra.run.smithi112.stdout:182 write 0x6dbdabd thru 0x6dc5cdf (0x8223 bytes) 2024-09-04T21:54:00.800 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.800 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.800 INFO:teuthology.orchestra.run.smithi112.stdout:187 writesame 0x8a46a11 thru 0x8a4df5c data_size 0x2(0x754c bytes) 2024-09-04T21:54:00.822 INFO:teuthology.orchestra.run.smithi112.stdout:190 punch from 0x78a9342 to 0x78b7d9f, (0xea5d bytes) 2024-09-04T21:54:00.822 INFO:teuthology.orchestra.run.smithi112.stdout:192 writesame 0x35997c4 thru 0x35a9609 data_size 0x39(0xfe46 bytes) 2024-09-04T21:54:00.824 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.824 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.829 INFO:teuthology.orchestra.run.smithi112.stdout:194 read 0x737ab85 thru 0x7385279 (0xa6f5 bytes) 2024-09-04T21:54:00.829 INFO:teuthology.orchestra.run.smithi112.stdout:195 punch from 0x14e88a0 to 0x14e89b5, (0x115 bytes) 2024-09-04T21:54:00.829 INFO:teuthology.orchestra.run.smithi112.stdout:196 write 0x49e7d74 thru 0x49edec4 (0x6151 bytes) 2024-09-04T21:54:00.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:00.844 INFO:teuthology.orchestra.run.smithi112.stdout:199 trunc from 0x964b657 to 0x6490860 2024-09-04T21:54:00.868 INFO:teuthology.orchestra.run.smithi112.stdout:200 punch from 0x61b8ec8 to 0x61bfb95, (0x6ccd bytes) 2024-09-04T21:54:00.875 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:00.876+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:00.900 INFO:teuthology.orchestra.run.smithi112.stdout:201 write 0xd22b513 thru 0xd23a27e (0xed6c bytes) 2024-09-04T21:54:00.907 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.907 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:207 punch from 0xa9bd65a to 0xa9bf89d, (0x2243 bytes) 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:208 read 0x4cadcd5 thru 0x4cb3704 (0x5a30 bytes) 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:209 read 0x427e102 thru 0x4285b32 (0x7a31 bytes) 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:210 read 0xbc3a24a thru 0xbc44a80 (0xa837 bytes) 2024-09-04T21:54:00.908 INFO:teuthology.orchestra.run.smithi112.stdout:211 write 0x2f9c34c thru 0x2fa6afe (0xa7b3 bytes) 2024-09-04T21:54:00.916 INFO:teuthology.orchestra.run.smithi112.stdout:215 write 0xade7126 thru 0xadec980 (0x585b bytes) 2024-09-04T21:54:00.925 INFO:teuthology.orchestra.run.smithi112.stdout:216 punch from 0x5cf7753 to 0x5d02c54, (0xb501 bytes) 2024-09-04T21:54:00.925 INFO:teuthology.orchestra.run.smithi112.stdout:218 punch from 0x49215ba to 0x492a725, (0x916b bytes) 2024-09-04T21:54:00.925 INFO:teuthology.orchestra.run.smithi112.stdout:219 writesame 0xab18796 thru 0xab1c8a4 data_size 0x1(0x410f bytes) 2024-09-04T21:54:00.938 INFO:teuthology.orchestra.run.smithi112.stdout:220 write 0x23270d0 thru 0x23338de (0xc80f bytes) 2024-09-04T21:54:00.946 INFO:teuthology.orchestra.run.smithi112.stdout:221 read 0xc256d55 thru 0xc259a55 (0x2d01 bytes) 2024-09-04T21:54:00.962 INFO:teuthology.orchestra.run.smithi112.stdout:222 trunc from 0xd23a27f to 0x719a5b9 2024-09-04T21:54:00.966 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:00.966 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:00.975 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:00.976+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:00.981 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:01.012 INFO:teuthology.orchestra.run.smithi112.stdout:224 write 0xce66d5d thru 0xce6d63c (0x68e0 bytes) 2024-09-04T21:54:01.017 INFO:teuthology.orchestra.run.smithi112.stdout:225 read 0x6f8fe4 thru 0x702a02 (0x9a1f bytes) 2024-09-04T21:54:01.017 INFO:teuthology.orchestra.run.smithi112.stdout:226 writesame 0xb4772a2 thru 0xb479799 data_size 0x8(0x24f8 bytes) 2024-09-04T21:54:01.022 INFO:teuthology.orchestra.run.smithi112.stdout:227 read 0x384450a thru 0x3852c29 (0xe720 bytes) 2024-09-04T21:54:01.023 INFO:teuthology.orchestra.run.smithi112.stdout:228 punch from 0x4aacc95 to 0x4abb598, (0xe903 bytes) 2024-09-04T21:54:01.023 INFO:teuthology.orchestra.run.smithi112.stdout:229 writesame 0xbb1069d thru 0xbb13093 data_size 0x1(0x29f7 bytes) 2024-09-04T21:54:01.034 INFO:teuthology.orchestra.run.smithi112.stdout:231 write 0x42d31a2 thru 0x42dea46 (0xb8a5 bytes) 2024-09-04T21:54:01.038 INFO:teuthology.orchestra.run.smithi112.stdout:233 read 0x36a4504 thru 0x36a4d8e (0x88b bytes) 2024-09-04T21:54:01.038 INFO:teuthology.orchestra.run.smithi112.stdout:235 writesame 0xc44d002 thru 0xc455198 data_size 0x5(0x8197 bytes) 2024-09-04T21:54:01.043 INFO:teuthology.orchestra.run.smithi157.stdout:2.7 2024-09-04T21:54:01.043 INFO:teuthology.orchestra.run.smithi157.stdout:2.0 2024-09-04T21:54:01.043 INFO:teuthology.orchestra.run.smithi157.stdout:2.5 2024-09-04T21:54:01.043 INFO:teuthology.orchestra.run.smithi157.stdout:1.0 2024-09-04T21:54:01.044 INFO:teuthology.orchestra.run.smithi112.stdout:236 write 0x482030a thru 0x482f2bc (0xefb3 bytes) 2024-09-04T21:54:01.072 INFO:teuthology.orchestra.run.smithi112.stdout:238 trunc from 0xce6d63d to 0xf1e9cf 2024-09-04T21:54:01.075 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:01.071+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:01.104 INFO:teuthology.orchestra.run.smithi112.stdout:239 trunc from 0xf1e9cf to 0x46eef95 2024-09-04T21:54:01.115 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:01.115 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:01.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:01.139 INFO:teuthology.orchestra.run.smithi112.stdout:242 write 0xbffcc66 thru 0xc002595 (0x5930 bytes) 2024-09-04T21:54:01.146 INFO:teuthology.orchestra.run.smithi112.stdout:243 write 0x728d86c thru 0x7290016 (0x27ab bytes) 2024-09-04T21:54:01.152 INFO:teuthology.orchestra.run.smithi112.stdout:244 write 0x73a38c0 thru 0x73b174e (0xde8f bytes) 2024-09-04T21:54:01.156 INFO:teuthology.orchestra.run.smithi112.stdout:245 read 0x471b11e thru 0x4728268 (0xd14b bytes) 2024-09-04T21:54:01.156 INFO:teuthology.orchestra.run.smithi112.stdout:247 writesame 0x8f02273 thru 0x8f10a36 data_size 0x1c(0xe7c4 bytes) 2024-09-04T21:54:01.161 INFO:teuthology.orchestra.run.smithi112.stdout:248 writesame 0x560fa3d thru 0x561936b data_size 0x1f(0x992f bytes) 2024-09-04T21:54:01.168 INFO:teuthology.orchestra.run.smithi112.stdout:249 write 0xabaa175 thru 0xabaacf5 (0xb81 bytes) 2024-09-04T21:54:01.177 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:01.176+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:01.184 INFO:teuthology.orchestra.run.smithi112.stdout:250 trunc from 0xc002596 to 0x387c7ed 2024-09-04T21:54:01.199 INFO:teuthology.orchestra.run.smithi112.stdout:251 punch from 0x1d34eaf to 0x1d4032d, (0xb47e bytes) 2024-09-04T21:54:01.199 INFO:teuthology.orchestra.run.smithi112.stdout:252 writesame 0xbd7d28 thru 0xbe5629 data_size 0x2(0xd902 bytes) 2024-09-04T21:54:01.224 INFO:teuthology.orchestra.run.smithi112.stdout:253 read 0x48a327 thru 0x49083f (0x6519 bytes) 2024-09-04T21:54:01.255 INFO:teuthology.orchestra.run.smithi112.stdout:255 write 0xd039365 thru 0xd03dc95 (0x4931 bytes) 2024-09-04T21:54:01.262 INFO:teuthology.orchestra.run.smithi112.stdout:256 read 0xc5d9997 thru 0xc5e9129 (0xf793 bytes) 2024-09-04T21:54:01.262 INFO:teuthology.orchestra.run.smithi112.stdout:258 read 0x848fcbc thru 0x849806e (0x83b3 bytes) 2024-09-04T21:54:01.262 INFO:teuthology.orchestra.run.smithi112.stdout:259 punch from 0x47047b7 to 0x470e011, (0x985a bytes) 2024-09-04T21:54:01.262 INFO:teuthology.orchestra.run.smithi112.stdout:261 write 0x4e6640 thru 0x4eec5d (0x861e bytes) 2024-09-04T21:54:01.270 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:01.270 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:01.277 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:01.276+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:01.279 INFO:teuthology.orchestra.run.smithi112.stdout:262 write 0xd781e66 thru 0xd784d64 (0x2eff bytes) 2024-09-04T21:54:01.286 INFO:teuthology.orchestra.run.smithi112.stdout:263 punch from 0x7fc1282 to 0x7fd0d18, (0xfa96 bytes) 2024-09-04T21:54:01.286 INFO:teuthology.orchestra.run.smithi112.stdout:264 read 0x32bf126 thru 0x32c2dab (0x3c86 bytes) 2024-09-04T21:54:01.286 INFO:teuthology.orchestra.run.smithi112.stdout:265 read 0x973a10c thru 0x973dde8 (0x3cdd bytes) 2024-09-04T21:54:01.286 INFO:teuthology.orchestra.run.smithi112.stdout:266 punch from 0x547831d to 0x548038e, (0x8071 bytes) 2024-09-04T21:54:01.287 INFO:teuthology.orchestra.run.smithi112.stdout:267 write 0xc322acd thru 0xc32890f (0x5e43 bytes) 2024-09-04T21:54:01.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:01.291 INFO:teuthology.orchestra.run.smithi112.stdout:268 writesame 0x89ac59c thru 0x89b3e10 data_size 0x13(0x7875 bytes) 2024-09-04T21:54:01.296 INFO:teuthology.orchestra.run.smithi112.stdout:269 read 0x4638db3 thru 0x463aee7 (0x2135 bytes) 2024-09-04T21:54:01.316 INFO:teuthology.orchestra.run.smithi112.stdout:270 trunc from 0xd784d65 to 0x1b04586 2024-09-04T21:54:01.326 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 2.7 --file /var/log/ceph/exp.2.7.7 2024-09-04T21:54:01.345 INFO:teuthology.orchestra.run.smithi112.stdout:271 trunc from 0x1b04586 to 0xc7188f2 2024-09-04T21:54:01.352 INFO:teuthology.orchestra.run.smithi112.stdout:272 read 0x2aa252c thru 0x2aaaa80 (0x8555 bytes) 2024-09-04T21:54:01.352 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.353 INFO:teuthology.orchestra.run.smithi112.stdout:276 write 0x2a23c85 thru 0x2a3385c (0xfbd8 bytes) 2024-09-04T21:54:01.357 INFO:teuthology.orchestra.run.smithi112.stdout:277 punch from 0x74e8246 to 0x74ecebe, (0x4c78 bytes) 2024-09-04T21:54:01.358 INFO:teuthology.orchestra.run.smithi112.stdout:278 write 0x910d472 thru 0x9116a8d (0x961c bytes) 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:279 read 0x26592bc thru 0x2659366 (0xab bytes) 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:282 read 0x4ea9115 thru 0x4eabe7f (0x2d6b bytes) 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:283 read 0xc26fde2 thru 0xc278eb0 (0x90cf bytes) 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:285 read 0xb770f0c thru 0xb773c2d (0x2d22 bytes) 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.363 INFO:teuthology.orchestra.run.smithi112.stdout:287 read 0x9899ea2 thru 0x989bc40 (0x1d9f bytes) 2024-09-04T21:54:01.364 INFO:teuthology.orchestra.run.smithi112.stdout:288 writesame 0x681c013 thru 0x6825cab data_size 0x17(0x9c99 bytes) 2024-09-04T21:54:01.368 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:54:01.372 INFO:teuthology.orchestra.run.smithi112.stdout:289 write 0x45e809c thru 0x45f325a (0xb1bf bytes) 2024-09-04T21:54:01.378 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:01.376+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:01.384 INFO:teuthology.orchestra.run.smithi112.stdout:290 write 0xccf6ff4 thru 0xcd01554 (0xa561 bytes) 2024-09-04T21:54:01.388 INFO:teuthology.orchestra.run.smithi112.stdout:292 read 0x2bea5ad thru 0x2bf1791 (0x71e5 bytes) 2024-09-04T21:54:01.388 INFO:teuthology.orchestra.run.smithi112.stdout:293 read 0x928ea76 thru 0x92951a7 (0x6732 bytes) 2024-09-04T21:54:01.389 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.389 INFO:teuthology.orchestra.run.smithi112.stdout:296 read 0x2625470 thru 0x262bfbc (0x6b4d bytes) 2024-09-04T21:54:01.389 INFO:teuthology.orchestra.run.smithi112.stdout:298 write 0x3fa056f thru 0x3fa9c93 (0x9725 bytes) 2024-09-04T21:54:01.394 INFO:teuthology.orchestra.run.smithi112.stdout:299 write 0x6f8e2d4 thru 0x6f8f0fc (0xe29 bytes) 2024-09-04T21:54:01.397 INFO:teuthology.orchestra.run.smithi112.stdout:300 read 0x906bd11 thru 0x90713b1 (0x56a1 bytes) 2024-09-04T21:54:01.398 INFO:teuthology.orchestra.run.smithi112.stdout:302 write 0x5371f1a thru 0x537e21b (0xc302 bytes) 2024-09-04T21:54:01.402 INFO:teuthology.orchestra.run.smithi112.stdout:303 read 0xb136325 thru 0xb13a300 (0x3fdc bytes) 2024-09-04T21:54:01.402 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.402 INFO:teuthology.orchestra.run.smithi112.stdout:305 punch from 0x95b23b4 to 0x95b57d9, (0x3425 bytes) 2024-09-04T21:54:01.402 INFO:teuthology.orchestra.run.smithi112.stdout:306 read 0x35c1d72 thru 0x35cebad (0xce3c bytes) 2024-09-04T21:54:01.409 INFO:teuthology.orchestra.run.smithi112.stdout:308 write 0xdb1b322 thru 0xdb20f10 (0x5bef bytes) 2024-09-04T21:54:01.412 INFO:teuthology.orchestra.run.smithi112.stdout:309 write 0xce32bbc thru 0xce42601 (0xfa46 bytes) 2024-09-04T21:54:01.425 INFO:teuthology.orchestra.run.smithi112.stdout:312 trunc from 0xdb20f11 to 0x760b71d 2024-09-04T21:54:01.434 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:01.434 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:01.449 INFO:teuthology.orchestra.run.smithi112.stdout:315 write 0xc8cccd2 thru 0xc8ce037 (0x1366 bytes) 2024-09-04T21:54:01.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:01.455 INFO:teuthology.orchestra.run.smithi112.stdout:316 write 0x1e93841 thru 0x1e9e8b5 (0xb075 bytes) 2024-09-04T21:54:01.461 INFO:teuthology.orchestra.run.smithi112.stdout:317 punch from 0x2a2446 to 0x2a382e, (0x13e8 bytes) 2024-09-04T21:54:01.461 INFO:teuthology.orchestra.run.smithi112.stdout:318 read 0x110f101 thru 0x111e4f5 (0xf3f5 bytes) 2024-09-04T21:54:01.461 INFO:teuthology.orchestra.run.smithi112.stdout:319 punch from 0x93bf74b to 0x93c9749, (0x9ffe bytes) 2024-09-04T21:54:01.461 INFO:teuthology.orchestra.run.smithi112.stdout:320 writesame 0x90bc956 thru 0x90c8bac data_size 0x2b(0xc257 bytes) 2024-09-04T21:54:01.476 INFO:teuthology.orchestra.run.smithi112.stdout:321 writesame 0xe37b054 thru 0xe38aa47 data_size 0x22(0xf9f4 bytes) 2024-09-04T21:54:01.476 INFO:teuthology.orchestra.run.smithi112.stdout:322 read 0x5d39971 thru 0x5d46ab6 (0xd146 bytes) 2024-09-04T21:54:01.476 INFO:teuthology.orchestra.run.smithi112.stdout:323 write 0x667f568 thru 0x668cb66 (0xd5ff bytes) 2024-09-04T21:54:01.478 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:01.476+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:01.483 INFO:teuthology.orchestra.run.smithi112.stdout:324 read 0x285b137 thru 0x286afa9 (0xfe73 bytes) 2024-09-04T21:54:01.484 INFO:teuthology.orchestra.run.smithi112.stdout:326 writesame 0x4fbd51f thru 0x4fc1239 data_size 0x1(0x3d1b bytes) 2024-09-04T21:54:01.495 INFO:teuthology.orchestra.run.smithi112.stdout:327 read 0x36dbd33 thru 0x36e476e (0x8a3c bytes) 2024-09-04T21:54:01.495 INFO:teuthology.orchestra.run.smithi112.stdout:329 punch from 0x8c881fd to 0x8c95c0e, (0xda11 bytes) 2024-09-04T21:54:01.495 INFO:teuthology.orchestra.run.smithi112.stdout:330 writesame 0x78ebf56 thru 0x78f386c data_size 0x1(0x7917 bytes) 2024-09-04T21:54:01.520 INFO:teuthology.orchestra.run.smithi112.stdout:331 write 0xe5bdf8a thru 0xe5c7b37 (0x9bae bytes) 2024-09-04T21:54:01.525 INFO:teuthology.orchestra.run.smithi112.stdout:332 write 0x8694cde thru 0x86a36f0 (0xea13 bytes) 2024-09-04T21:54:01.531 INFO:teuthology.orchestra.run.smithi112.stdout:334 write 0xb6610ba thru 0xb66a040 (0x8f87 bytes) 2024-09-04T21:54:01.541 INFO:teuthology.orchestra.run.smithi112.stdout:335 trunc from 0xe5c7b38 to 0xbfd314f 2024-09-04T21:54:01.555 INFO:teuthology.orchestra.run.smithi112.stdout:336 write 0x297f9bf thru 0x2980b7d (0x11bf bytes) 2024-09-04T21:54:01.558 INFO:teuthology.orchestra.run.smithi112.stdout:338 write 0x31dfe69 thru 0x31e7fc5 (0x815d bytes) 2024-09-04T21:54:01.562 INFO:teuthology.orchestra.run.smithi112.stdout:339 read 0x6178e5a thru 0x617afed (0x2194 bytes) 2024-09-04T21:54:01.562 INFO:teuthology.orchestra.run.smithi112.stdout:340 punch from 0x5dd97 to 0x5ea90, (0xcf9 bytes) 2024-09-04T21:54:01.562 INFO:teuthology.orchestra.run.smithi112.stdout:342 punch from 0x5d77df7 to 0x5d841fe, (0xc407 bytes) 2024-09-04T21:54:01.562 INFO:teuthology.orchestra.run.smithi112.stdout:344 read 0xad7c90e thru 0xad871ea (0xa8dd bytes) 2024-09-04T21:54:01.562 INFO:teuthology.orchestra.run.smithi112.stdout:345 write 0x891a745 thru 0x892096f (0x622b bytes) 2024-09-04T21:54:01.567 INFO:teuthology.orchestra.run.smithi112.stdout:346 read 0x85569bb thru 0x855a15b (0x37a1 bytes) 2024-09-04T21:54:01.567 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:01.568 INFO:teuthology.orchestra.run.smithi112.stdout:347 read 0x9418250 thru 0x9427bcb (0xf97c bytes) 2024-09-04T21:54:01.569 INFO:teuthology.orchestra.run.smithi112.stdout:348 write 0x13996d7 thru 0x139df8b (0x48b5 bytes) 2024-09-04T21:54:01.572 INFO:teuthology.orchestra.run.smithi112.stdout:349 read 0x7db80e9 thru 0x7dbb9c5 (0x38dd bytes) 2024-09-04T21:54:01.573 INFO:teuthology.orchestra.run.smithi112.stdout:351 punch from 0x3f7a09c to 0x3f7d634, (0x3598 bytes) 2024-09-04T21:54:01.573 INFO:teuthology.orchestra.run.smithi112.stdout:353 write 0x622535f thru 0x622695f (0x1601 bytes) 2024-09-04T21:54:01.578 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.579 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.579 INFO:teuthology.orchestra.run.smithi112.stdout:357 read 0x6e432c thru 0x6f2c34 (0xe909 bytes) 2024-09-04T21:54:01.579 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:01.580+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:01.579 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.579 INFO:teuthology.orchestra.run.smithi112.stdout:359 write 0x68bb3f6 thru 0x68c1871 (0x647c bytes) 2024-09-04T21:54:01.581 INFO:teuthology.orchestra.run.smithi112.stdout:360 writesame 0x4662ddf thru 0x466bf14 data_size 0x1(0x9136 bytes) 2024-09-04T21:54:01.590 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:01.590 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:01.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:01.614 INFO:teuthology.orchestra.run.smithi112.stdout:361 read 0x8fb0bf3 thru 0x8fb10a6 (0x4b4 bytes) 2024-09-04T21:54:01.624 INFO:teuthology.orchestra.run.smithi112.stdout:362 writesame 0xc4b071e thru 0xc4b508e data_size 0x3(0x4971 bytes) 2024-09-04T21:54:01.624 INFO:teuthology.orchestra.run.smithi112.stdout:365 read 0x5dcf915 thru 0x5dd5b51 (0x623d bytes) 2024-09-04T21:54:01.624 INFO:teuthology.orchestra.run.smithi112.stdout:366 write 0x5a202ad thru 0x5a20594 (0x2e8 bytes) 2024-09-04T21:54:01.628 INFO:teuthology.orchestra.run.smithi112.stdout:367 writesame 0xa8d1a2f thru 0xa8d6b54 data_size 0x2f(0x5126 bytes) 2024-09-04T21:54:01.633 INFO:teuthology.orchestra.run.smithi112.stdout:370 write 0x537f09 thru 0x5474db (0xf5d3 bytes) 2024-09-04T21:54:01.637 INFO:teuthology.orchestra.run.smithi112.stdout:372 writesame 0x61f051a thru 0x61fa304 data_size 0x1(0x9deb bytes) 2024-09-04T21:54:01.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-04T21:54:01.662 INFO:teuthology.orchestra.run.smithi112.stdout:373 read 0xbbe5d6b thru 0xbbf0fac (0xb242 bytes) 2024-09-04T21:54:01.662 INFO:teuthology.orchestra.run.smithi112.stdout:377 writesame 0xb8dd6f2 thru 0xb8dd858 data_size 0x1(0x167 bytes) 2024-09-04T21:54:01.667 INFO:teuthology.orchestra.run.smithi112.stdout:379 read 0xac5ecb2 thru 0xac6ce09 (0xe158 bytes) 2024-09-04T21:54:01.667 INFO:teuthology.orchestra.run.smithi112.stdout:380 writesame 0x596db4f thru 0x5977633 data_size 0x1(0x9ae5 bytes) 2024-09-04T21:54:01.680 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:01.675+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:01.697 INFO:teuthology.orchestra.run.smithi112.stdout:381 trunc from 0xc4b508f to 0x63b68dc 2024-09-04T21:54:01.725 INFO:teuthology.orchestra.run.smithi112.stdout:382 writesame 0x79b69e7 thru 0x79bcc83 data_size 0x3(0x629d bytes) 2024-09-04T21:54:01.733 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:01.739 INFO:teuthology.orchestra.run.smithi112.stdout:383 write 0xc85a998 thru 0xc85b097 (0x700 bytes) 2024-09-04T21:54:01.743 INFO:teuthology.orchestra.run.smithi112.stdout:386 read 0x2e7fbcb thru 0x2e814bf (0x18f5 bytes) 2024-09-04T21:54:01.743 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.743 INFO:teuthology.orchestra.run.smithi112.stdout:388 write 0x8bcf08a thru 0x8bdefd1 (0xff48 bytes) 2024-09-04T21:54:01.748 INFO:teuthology.orchestra.run.smithi112.stdout:389 punch from 0x60c64fe to 0x60d5081, (0xeb83 bytes) 2024-09-04T21:54:01.749 INFO:teuthology.orchestra.run.smithi112.stdout:390 read 0x47b2861 thru 0x47b2fdb (0x77b bytes) 2024-09-04T21:54:01.753 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:01.753 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:01.755 INFO:teuthology.orchestra.run.smithi112.stdout:392 trunc from 0xc85b098 to 0x918f9f2 2024-09-04T21:54:01.755 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:01.755 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:01.755 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:01.755 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:01.755 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:01.768 INFO:teuthology.orchestra.run.smithi112.stdout:393 read 0x70a4251 thru 0x70a8e18 (0x4bc8 bytes) 2024-09-04T21:54:01.768 INFO:teuthology.orchestra.run.smithi112.stdout:394 writesame 0x894c185 thru 0x8951b2d data_size 0x7(0x59a9 bytes) 2024-09-04T21:54:01.771 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:01.772 INFO:teuthology.orchestra.run.smithi112.stdout:395 read 0x2537f40 thru 0x25421bd (0xa27e bytes) 2024-09-04T21:54:01.773 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:01.773 INFO:teuthology.orchestra.run.smithi112.stdout:396 trunc from 0x918f9f2 to 0x99815a6 2024-09-04T21:54:01.781 INFO:teuthology.orchestra.run.smithi112.stdout:397 read 0x470e9ae thru 0x471760d (0x8c60 bytes) 2024-09-04T21:54:01.781 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:01.775+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:01.792 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:01.792 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:01.794 INFO:teuthology.orchestra.run.smithi112.stdout:398 write 0xd8c8f20 thru 0xd8d737d (0xe45e bytes) 2024-09-04T21:54:01.798 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.798 INFO:teuthology.orchestra.run.smithi112.stdout:400 read 0xa2e0ff3 thru 0xa2efdcb (0xedd9 bytes) 2024-09-04T21:54:01.798 INFO:teuthology.orchestra.run.smithi112.stdout:401 write 0x6be4d35 thru 0x6bed7a9 (0x8a75 bytes) 2024-09-04T21:54:01.801 INFO:teuthology.orchestra.run.smithi112.stdout:402 read 0x68208e0 thru 0x6821f13 (0x1634 bytes) 2024-09-04T21:54:01.802 INFO:teuthology.orchestra.run.smithi112.stdout:403 writesame 0x2a46200 thru 0x2a4b125 data_size 0x3(0x4f26 bytes) 2024-09-04T21:54:01.805 INFO:teuthology.orchestra.run.smithi112.stdout:404 write 0x498eaeb thru 0x499b2c4 (0xc7da bytes) 2024-09-04T21:54:01.809 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.809 INFO:teuthology.orchestra.run.smithi112.stdout:406 writesame 0x485d8ca thru 0x485fab3 data_size 0x3(0x21ea bytes) 2024-09-04T21:54:01.811 INFO:teuthology.orchestra.run.smithi112.stdout:407 punch from 0x361ca09 to 0x362acdf, (0xe2d6 bytes) 2024-09-04T21:54:01.811 INFO:teuthology.orchestra.run.smithi112.stdout:409 read 0x1992fcd thru 0x19944a9 (0x14dd bytes) 2024-09-04T21:54:01.811 INFO:teuthology.orchestra.run.smithi112.stdout:410 read 0xc1137fa thru 0xc11ef7a (0xb781 bytes) 2024-09-04T21:54:01.811 INFO:teuthology.orchestra.run.smithi112.stdout:411 punch from 0x2d9b56f to 0x2da044d, (0x4ede bytes) 2024-09-04T21:54:01.811 INFO:teuthology.orchestra.run.smithi112.stdout:414 write 0xb4a26d5 thru 0xb4b146d (0xed99 bytes) 2024-09-04T21:54:01.812 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:01.812 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:01.814 INFO:teuthology.orchestra.run.smithi112.stdout:416 write 0xbdc3bb9 thru 0xbdc420e (0x656 bytes) 2024-09-04T21:54:01.819 INFO:teuthology.orchestra.run.smithi112.stdout:418 read 0x84a36f1 thru 0x84a7935 (0x4245 bytes) 2024-09-04T21:54:01.819 INFO:teuthology.orchestra.run.smithi112.stdout:419 read 0x5e5f906 thru 0x5e69d1f (0xa41a bytes) 2024-09-04T21:54:01.819 INFO:teuthology.orchestra.run.smithi112.stdout:420 punch from 0x8d96457 to 0x8d98640, (0x21e9 bytes) 2024-09-04T21:54:01.819 INFO:teuthology.orchestra.run.smithi112.stdout:421 write 0x28a2b99 thru 0x28b153f (0xe9a7 bytes) 2024-09-04T21:54:01.820 INFO:teuthology.orchestra.run.smithi112.stdout:422 read 0x408050b thru 0x4080bc7 (0x6bd bytes) 2024-09-04T21:54:01.820 INFO:teuthology.orchestra.run.smithi112.stdout:423 read 0x32f99e7 thru 0x33046b4 (0xacce bytes) 2024-09-04T21:54:01.821 INFO:teuthology.orchestra.run.smithi112.stdout:424 writesame 0xc4efc72 thru 0xc4f6932 data_size 0x1(0x6cc1 bytes) 2024-09-04T21:54:01.822 INFO:teuthology.orchestra.run.smithi157.stderr:Exporting 2.7 info 2.7( empty local-lis/les=18/19 n=0 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-09-04T21:54:01.823 INFO:teuthology.orchestra.run.smithi157.stderr:Export successful 2024-09-04T21:54:01.830 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:01.831 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:01.832 INFO:teuthology.orchestra.run.smithi112.stdout:425 write 0xba27177 thru 0xba28b71 (0x19fb bytes) 2024-09-04T21:54:01.835 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.836 INFO:teuthology.orchestra.run.smithi112.stdout:431 read 0xcaae344 thru 0xcabe175 (0xfe32 bytes) 2024-09-04T21:54:01.836 INFO:teuthology.orchestra.run.smithi112.stdout:435 read 0x96a124c thru 0x96a271c (0x14d1 bytes) 2024-09-04T21:54:01.836 INFO:teuthology.orchestra.run.smithi112.stdout:436 read 0x4eb0762 thru 0x4eb1929 (0x11c8 bytes) 2024-09-04T21:54:01.836 INFO:teuthology.orchestra.run.smithi112.stdout:437 read 0x3cf93fb thru 0x3d093cf (0xffd5 bytes) 2024-09-04T21:54:01.837 INFO:teuthology.orchestra.run.smithi112.stdout:438 write 0xa5669f thru 0xa65410 (0xed72 bytes) 2024-09-04T21:54:01.838 INFO:teuthology.orchestra.run.smithi112.stdout:439 write 0xd4dab8a thru 0xd4e8808 (0xdc7f bytes) 2024-09-04T21:54:01.842 INFO:teuthology.orchestra.run.smithi112.stdout:441 write 0x2599313 thru 0x25a16ca (0x83b8 bytes) 2024-09-04T21:54:01.846 INFO:teuthology.orchestra.run.smithi112.stdout:444 write 0x71d42a9 thru 0x71e2299 (0xdff1 bytes) 2024-09-04T21:54:01.850 INFO:teuthology.orchestra.run.smithi112.stdout:445 write 0x40534ba thru 0x40593d0 (0x5f17 bytes) 2024-09-04T21:54:01.853 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:01.853 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:01.853 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:01.854 INFO:teuthology.orchestra.run.smithi112.stdout:446 write 0x199615b thru 0x199ecc5 (0x8b6b bytes) 2024-09-04T21:54:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:01.857 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:01.864 INFO:teuthology.orchestra.run.smithi112.stdout:447 write 0xdf34a60 thru 0xdf38a0e (0x3faf bytes) 2024-09-04T21:54:01.868 INFO:teuthology.orchestra.run.smithi112.stdout:450 write 0xbe09950 thru 0xbe14599 (0xac4a bytes) 2024-09-04T21:54:01.871 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:01.871 INFO:teuthology.orchestra.run.smithi112.stdout:453 writesame 0x3edc4fc thru 0x3eeb9e9 data_size 0x6b(0xf4ee bytes) 2024-09-04T21:54:01.873 INFO:teuthology.orchestra.run.smithi112.stdout:454 write 0x205474c thru 0x2057501 (0x2db6 bytes) 2024-09-04T21:54:01.877 INFO:teuthology.orchestra.run.smithi112.stdout:455 writesame 0x84e3868 thru 0x84e7510 data_size 0x1(0x3ca9 bytes) 2024-09-04T21:54:01.881 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:01.880+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:01.884 INFO:teuthology.orchestra.run.smithi112.stdout:456 punch from 0xc864516 to 0xc87048c, (0xbf76 bytes) 2024-09-04T21:54:01.884 INFO:teuthology.orchestra.run.smithi112.stdout:458 read 0x93d2322 thru 0x93d86f5 (0x63d4 bytes) 2024-09-04T21:54:01.884 INFO:teuthology.orchestra.run.smithi112.stdout:459 punch from 0x33490df to 0x334dd45, (0x4c66 bytes) 2024-09-04T21:54:01.899 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:01.899 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4208.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:53:46.809667+0000", 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.092619596, 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020134630000000001, 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:01.900 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809667+0000", 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809667+0000", 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809667+0000", 2024-09-04T21:54:01.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5999999999999999e-07 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809682+0000", 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.464e-05 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809707+0000", 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4817e-05 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809723+0000", 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6405999999999998e-05 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809857+0000", 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013378199999999999 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.809909+0000", 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2320000000000001e-05 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.811596+0000", 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016867519999999999 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:46.811680+0000", 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4285999999999999e-05 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.904 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4208.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:53:47.815151+0000", 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.087135715000001, 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017320149999999999, 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815151+0000", 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:01.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815151+0000", 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815151+0000", 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999995e-07 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:01.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815160+0000", 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3330000000000003e-06 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815181+0000", 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0789999999999999e-05 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815206+0000", 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4655999999999999e-05 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815366+0000", 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000160545 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.815422+0000", 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-05 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.816796+0000", 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001373452 2024-09-04T21:54:01.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:47.816883+0000", 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6731999999999996e-05 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:01.909 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:01.910 INFO:teuthology.orchestra.run.smithi112.stdout:460 trunc from 0xdf38a0f to 0x130a807 2024-09-04T21:54:01.919 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:461 read 0xd93155 thru 0xd98b4b (0x59f7 bytes) 2024-09-04T21:54:01.929 INFO:teuthology.orchestra.run.smithi112.stdout:462 punch from 0x1280928 to 0x1284a2c, (0x4104 bytes) 2024-09-04T21:54:01.929 INFO:teuthology.orchestra.run.smithi112.stdout:463 clone 1 order 19 su 32768 sc 5 2024-09-04T21:54:01.983 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:01.980+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:02.042 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:02.043 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:02.043 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:02.043 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:02.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:02.099 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:02.100+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:02.117 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 2.7 2024-09-04T21:54:02.185 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:02.184+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:02.215 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:02.215 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:02.215 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:02.215 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:02.216 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:02.229 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:02.315 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:02.316+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:02.386 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:02.384+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:02.401 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:02.401 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:02.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:02.401 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:02.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.402 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4208.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-04T21:54:02.402 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:53:45.799169+0000", 2024-09-04T21:54:02.402 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.603208091999999, 2024-09-04T21:54:02.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014152399999999999, 2024-09-04T21:54:02.402 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.403 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799169+0000", 2024-09-04T21:54:02.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799169+0000", 2024-09-04T21:54:02.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799170+0000", 2024-09-04T21:54:02.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2999999999999999e-07 2024-09-04T21:54:02.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799192+0000", 2024-09-04T21:54:02.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2033999999999999e-05 2024-09-04T21:54:02.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799250+0000", 2024-09-04T21:54:02.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8322000000000002e-05 2024-09-04T21:54:02.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799273+0000", 2024-09-04T21:54:02.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.27e-05 2024-09-04T21:54:02.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799396+0000", 2024-09-04T21:54:02.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122915 2024-09-04T21:54:02.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.799439+0000", 2024-09-04T21:54:02.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2945000000000003e-05 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.800505+0000", 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001066474 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:53:45.800584+0000", 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9019999999999999e-05 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:985 3.4 e27/22 3:25c933e3:::rbd_data.11882e693cd0.000000000000002d:head v 27'7, mlcod=27'5)", 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.816419+0000", 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.58595777599999999, 2024-09-04T21:54:02.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001330896, 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816419+0000", 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816419+0000", 2024-09-04T21:54:02.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816421+0000", 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6279999999999999e-06 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816452+0000", 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0953e-05 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816494+0000", 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2029e-05 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816505+0000", 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0755e-05 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816577+0000", 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1853999999999993e-05 2024-09-04T21:54:02.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.816603+0000", 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.616e-05 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.817687+0000", 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010844050000000001 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.817750+0000", 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3112000000000002e-05 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1036 3.0 e27/22 3:029d3929:::rbd_data.11882e693cd0.0000000000000014:head v 27'13, mlcod=26'11)", 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.911238+0000", 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.491138939, 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00080471199999999996, 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.418 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911238+0000", 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911238+0000", 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911240+0000", 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7379999999999999e-06 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911249+0000", 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5070000000000006e-06 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911268+0000", 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8481000000000001e-05 2024-09-04T21:54:02.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911276+0000", 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3469999999999999e-06 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911352+0000", 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5613999999999998e-05 2024-09-04T21:54:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911374+0000", 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1977e-05 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.912000+0000", 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062569199999999996 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.912043+0000", 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3356000000000003e-05 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1043 3.4 e27/22 3:25c933e3:::rbd_data.11882e693cd0.000000000000002d:head v 27'8, mlcod=27'6)", 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.913871+0000", 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.48850652900000002, 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001939054, 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913871+0000", 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913871+0000", 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913872+0000", 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.176e-06 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913879+0000", 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0650000000000001e-06 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913892+0000", 2024-09-04T21:54:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3672999999999999e-05 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913901+0000", 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0530000000000007e-06 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913960+0000", 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9688000000000002e-05 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913983+0000", 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2605000000000001e-05 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.915763+0000", 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001780535 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.915810+0000", 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6258999999999999e-05 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.428 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:02.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:02.519 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:02.520+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:02.539 INFO:teuthology.orchestra.run.smithi112.stdout:464 trunc from 0x130a807 to 0x852218b 2024-09-04T21:54:02.576 INFO:teuthology.orchestra.run.smithi112.stdout:465 write 0xe19ffe2 thru 0xe1aebcb (0xebea bytes) 2024-09-04T21:54:02.585 INFO:teuthology.orchestra.run.smithi112.stdout:467 read 0x2e98956 thru 0x2ea622a (0xd8d5 bytes) 2024-09-04T21:54:02.585 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.585 INFO:teuthology.orchestra.run.smithi112.stdout:469 write 0x7f33b5c thru 0x7f4173c (0xdbe1 bytes) 2024-09-04T21:54:02.588 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:02.588+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:02.602 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:02.602 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:02.602 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:02.603 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:02.620 INFO:teuthology.orchestra.run.smithi112.stdout:470 trunc from 0xe1aebcc to 0x2899ef8 2024-09-04T21:54:02.624 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:02.632 INFO:teuthology.orchestra.run.smithi112.stdout:471 read 0x60ffa8 thru 0x61f47e (0xf4d7 bytes) 2024-09-04T21:54:02.632 INFO:teuthology.orchestra.run.smithi112.stdout:475 read 0x146875b thru 0x146a27b (0x1b21 bytes) 2024-09-04T21:54:02.632 INFO:teuthology.orchestra.run.smithi112.stdout:476 punch from 0x1b3b5f0 to 0x1b3e3f0, (0x2e00 bytes) 2024-09-04T21:54:02.633 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.633 INFO:teuthology.orchestra.run.smithi112.stdout:478 read 0x5925d0 thru 0x5975f6 (0x5027 bytes) 2024-09-04T21:54:02.644 INFO:teuthology.orchestra.run.smithi112.stdout:479 trunc from 0x2899ef8 to 0x99e00ef 2024-09-04T21:54:02.651 INFO:teuthology.orchestra.run.smithi112.stdout:481 read 0x16c99dc thru 0x16d9288 (0xf8ad bytes) 2024-09-04T21:54:02.651 INFO:teuthology.orchestra.run.smithi112.stdout:482 write 0x81c5231 thru 0x81d5094 (0xfe64 bytes) 2024-09-04T21:54:02.662 INFO:teuthology.orchestra.run.smithi112.stdout:483 writesame 0x86f4555 thru 0x8701b64 data_size 0x89(0xd610 bytes) 2024-09-04T21:54:02.667 INFO:teuthology.orchestra.run.smithi112.stdout:484 read 0x5d50397 thru 0x5d5a2a6 (0x9f10 bytes) 2024-09-04T21:54:02.667 INFO:teuthology.orchestra.run.smithi112.stdout:485 write 0x5ee815a thru 0x5ef2f61 (0xae08 bytes) 2024-09-04T21:54:02.672 INFO:teuthology.orchestra.run.smithi112.stdout:486 read 0x62ad28b thru 0x62b14b4 (0x422a bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:487 read 0x34e9625 thru 0x34ee752 (0x512e bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:489 punch from 0x44cf216 to 0x44d7e8e, (0x8c78 bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:492 punch from 0x7588793 to 0x7591900, (0x916d bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:493 punch from 0x9c34cb to 0x9c7506, (0x403b bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:495 read 0x82bee5b thru 0x82c73e3 (0x8589 bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:497 read 0x4d7d656 thru 0x4d832c1 (0x5c6c bytes) 2024-09-04T21:54:02.673 INFO:teuthology.orchestra.run.smithi112.stdout:498 writesame 0x4418f87 thru 0x4428a5b data_size 0x1(0xfad5 bytes) 2024-09-04T21:54:02.688 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:02.688+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:02.695 INFO:teuthology.orchestra.run.smithi112.stdout:499 write 0x7aab24e thru 0x7aad25f (0x2012 bytes) 2024-09-04T21:54:02.713 INFO:teuthology.orchestra.run.smithi112.stdout:500 write 0xc571ff5 thru 0xc57b181 (0x918d bytes) 2024-09-04T21:54:02.738 INFO:teuthology.orchestra.run.smithi112.stdout:501 writesame 0xe8a4458 thru 0xe8b1ce0 data_size 0x1(0xd889 bytes) 2024-09-04T21:54:02.738 INFO:teuthology.orchestra.run.smithi112.stdout:502 read 0x3575de9 thru 0x357ce06 (0x701e bytes) 2024-09-04T21:54:02.738 INFO:teuthology.orchestra.run.smithi112.stdout:503 punch from 0xb9241f3 to 0xb931d83, (0xdb90 bytes) 2024-09-04T21:54:02.750 INFO:teuthology.orchestra.run.smithi112.stdout:504 trunc from 0xe8b1ce1 to 0x8b2076c 2024-09-04T21:54:02.759 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:02.759 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:02.759 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:02.759 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:02.759 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:02.774 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:02.775 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.776 INFO:teuthology.orchestra.run.smithi112.stdout:506 read 0x231cf27 thru 0x231e213 (0x12ed bytes) 2024-09-04T21:54:02.776 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.776 INFO:teuthology.orchestra.run.smithi112.stdout:508 punch from 0x7f9aa83 to 0x7fa87c7, (0xdd44 bytes) 2024-09-04T21:54:02.776 INFO:teuthology.orchestra.run.smithi112.stdout:510 read 0x5e97358 thru 0x5e98fdd (0x1c86 bytes) 2024-09-04T21:54:02.776 INFO:teuthology.orchestra.run.smithi112.stdout:511 write 0x1826390 thru 0x1829edd (0x3b4e bytes) 2024-09-04T21:54:02.785 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.789 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:02.788+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:02.792 INFO:teuthology.orchestra.run.smithi112.stdout:513 trunc from 0x8b2076c to 0xd5f1c05 2024-09-04T21:54:02.803 INFO:teuthology.orchestra.run.smithi112.stdout:514 read 0xcce21c thru 0xccf14e (0xf33 bytes) 2024-09-04T21:54:02.804 INFO:teuthology.orchestra.run.smithi112.stdout:515 read 0x28bbd93 thru 0x28c08a9 (0x4b17 bytes) 2024-09-04T21:54:02.805 INFO:teuthology.orchestra.run.smithi112.stdout:516 read 0xa311560 thru 0xa31f1e7 (0xdc88 bytes) 2024-09-04T21:54:02.805 INFO:teuthology.orchestra.run.smithi112.stdout:517 punch from 0x239a99f to 0x239eb25, (0x4186 bytes) 2024-09-04T21:54:02.805 INFO:teuthology.orchestra.run.smithi112.stdout:519 punch from 0x316d22d to 0x31785d2, (0xb3a5 bytes) 2024-09-04T21:54:02.805 INFO:teuthology.orchestra.run.smithi112.stdout:523 write 0xc77de67 thru 0xc78a54f (0xc6e9 bytes) 2024-09-04T21:54:02.814 INFO:teuthology.orchestra.run.smithi112.stdout:525 punch from 0x508c0b2 to 0x509769c, (0xb5ea bytes) 2024-09-04T21:54:02.814 INFO:teuthology.orchestra.run.smithi112.stdout:526 read 0x688eb78 thru 0x6893f9e (0x5427 bytes) 2024-09-04T21:54:02.814 INFO:teuthology.orchestra.run.smithi112.stdout:528 write 0x35c3729 thru 0x35c4c95 (0x156d bytes) 2024-09-04T21:54:02.818 INFO:teuthology.orchestra.run.smithi112.stdout:529 read 0xac5558a thru 0xac584db (0x2f52 bytes) 2024-09-04T21:54:02.818 INFO:teuthology.orchestra.run.smithi112.stdout:530 write 0x470f88b thru 0x471c2f1 (0xca67 bytes) 2024-09-04T21:54:02.825 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.825 INFO:teuthology.orchestra.run.smithi112.stdout:533 punch from 0xb59619b to 0xb59db45, (0x79aa bytes) 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:534 read 0x8b2be69 thru 0x8b32e9b (0x7033 bytes) 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:535 read 0x5d647b9 thru 0x5d6c9b5 (0x81fd bytes) 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:536 read 0xc0381dc thru 0xc03beb2 (0x3cd7 bytes) 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:537 read 0x5b4fa1 thru 0x5c248b (0xd4eb bytes) 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.826 INFO:teuthology.orchestra.run.smithi112.stdout:544 read 0xbd55a69 thru 0xbd5f0a8 (0x9640 bytes) 2024-09-04T21:54:02.827 INFO:teuthology.orchestra.run.smithi112.stdout:545 read 0x5d5a412 thru 0x5d68ff8 (0xebe7 bytes) 2024-09-04T21:54:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:547 trunc from 0xd5f1c05 to 0x21ad5d2 2024-09-04T21:54:02.868 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.868 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.868 INFO:teuthology.orchestra.run.smithi112.stdout:554 read 0x9b7a69 thru 0x9c3ccb (0xc263 bytes) 2024-09-04T21:54:02.884 INFO:teuthology.orchestra.run.smithi112.stdout:555 trunc from 0x21ad5d2 to 0xa77f995 2024-09-04T21:54:02.890 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:02.887+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:02.893 INFO:teuthology.orchestra.run.smithi112.stdout:556 read 0x9e9c8d9 thru 0x9ea6d8c (0xa4b4 bytes) 2024-09-04T21:54:02.893 INFO:teuthology.orchestra.run.smithi112.stdout:557 write 0x6d0c000 thru 0x6d0f333 (0x3334 bytes) 2024-09-04T21:54:02.897 INFO:teuthology.orchestra.run.smithi112.stdout:558 read 0x9ce33a8 thru 0x9cec448 (0x90a1 bytes) 2024-09-04T21:54:02.897 INFO:teuthology.orchestra.run.smithi112.stdout:559 write 0x49f681d thru 0x4a05fcb (0xf7af bytes) 2024-09-04T21:54:02.903 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.903 INFO:teuthology.orchestra.run.smithi112.stdout:561 read 0xa5628d0 thru 0xa56bbb6 (0x92e7 bytes) 2024-09-04T21:54:02.903 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.903 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:02.912 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:02.912 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1025 3.1e 3:7c20971b:::rbd_data.11882e693cd0.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.907792+0000", 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.0033377269999999, 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003035486, 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1025 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907792+0000", 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907792+0000", 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907793+0000", 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3099999999999996e-07 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907797+0000", 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1810000000000003e-06 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907809+0000", 2024-09-04T21:54:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1218e-05 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907816+0000", 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0890000000000002e-06 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907870+0000", 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4055999999999999e-05 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.907947+0000", 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.908075+0000", 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012857600000000001 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.908634+0000", 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055861699999999997 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.910755+0000", 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002120739 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.910790+0000", 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5710000000000002e-05 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.910828+0000", 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7373999999999998e-05 2024-09-04T21:54:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1037 3.2 3:459c71c3:::rbd_data.11882e693cd0.0000000000000016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.911334+0000", 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.99979660599999998, 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023732940000000002, 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1037 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911334+0000", 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911334+0000", 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911335+0000", 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2979999999999999e-06 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911340+0000", 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6450000000000002e-06 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911354+0000", 2024-09-04T21:54:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3749e-05 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911362+0000", 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9859999999999994e-06 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911425+0000", 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3019000000000001e-05 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911516+0000", 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1026999999999994e-05 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911632+0000", 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011579299999999999 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.912490+0000", 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085757900000000002 2024-09-04T21:54:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913603+0000", 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011130039999999999 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913666+0000", 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3454999999999995e-05 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913707+0000", 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0738999999999997e-05 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1038 3.1e 3:78f6dc0f:::rbd_data.11882e693cd0.0000000000000018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.911640+0000", 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.99949029599999994, 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029039589999999998, 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1038 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911640+0000", 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911640+0000", 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911641+0000", 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.78e-07 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911645+0000", 2024-09-04T21:54:02.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6979999999999997e-06 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911656+0000", 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911664+0000", 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.435e-06 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911759+0000", 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5419000000000002e-05 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.911931+0000", 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000171838 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.912132+0000", 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020115199999999999 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.914088+0000", 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019560340000000002 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.914437+0000", 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034925899999999998 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.914500+0000", 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2557000000000004e-05 2024-09-04T21:54:02.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.914544+0000", 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3772e-05 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1042 3.1e 3:7886ef8d:::rbd_data.11882e693cd0.0000000000000022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.913157+0000", 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.99797279400000005, 2024-09-04T21:54:02.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032261019999999998, 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1042 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913157+0000", 2024-09-04T21:54:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913157+0000", 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913159+0000", 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.274e-06 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913171+0000", 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.278e-05 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913185+0000", 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3927000000000001e-05 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913193+0000", 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7190000000000001e-06 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913308+0000", 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011503299999999999 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913402+0000", 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4032999999999999e-05 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913615+0000", 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021307100000000001 2024-09-04T21:54:02.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.915787+0000", 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021722450000000002 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.916308+0000", 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00052021700000000001 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.916349+0000", 2024-09-04T21:54:02.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1433000000000001e-05 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.916383+0000", 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4369999999999998e-05 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1044 3.8 3:11b7ccdb:::rbd_data.11882e693cd0.000000000000002e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.913330+0000", 2024-09-04T21:54:02.937 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.99779994299999997, 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053603039999999998, 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1044 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913330+0000", 2024-09-04T21:54:02.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913330+0000", 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913331+0000", 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.091e-06 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913341+0000", 2024-09-04T21:54:02.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0212e-05 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913353+0000", 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1155000000000001e-05 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913360+0000", 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6580000000000002e-06 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913443+0000", 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2288999999999998e-05 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913567+0000", 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000123943 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913783+0000", 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021682 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.915863+0000", 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020799 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.918587+0000", 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027240400000000001 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.918644+0000", 2024-09-04T21:54:02.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6436e-05 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.918690+0000", 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.676e-05 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1050 3.8 3:121ca1cb:::rbd_object_map.11882e693cd0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.919214+0000", 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.99191644700000003, 2024-09-04T21:54:02.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021625059999999998, 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919214+0000", 2024-09-04T21:54:02.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919214+0000", 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919215+0000", 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7e-06 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919223+0000", 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2520000000000002e-06 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919240+0000", 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7560000000000001e-05 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919250+0000", 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8500000000000006e-06 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919308+0000", 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7893999999999998e-05 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919380+0000", 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2498999999999998e-05 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.919634+0000", 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025318200000000002 2024-09-04T21:54:02.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921244+0000", 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001610268 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921315+0000", 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1328000000000006e-05 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921347+0000", 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2067000000000003e-05 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921376+0000", 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8906000000000001e-05 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1051 3.15 3:ac2f3301:::rbd_data.11882e693cd0.0000000000000004:head [truncate 3188743] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.921549+0000", 2024-09-04T21:54:02.949 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.98958075300000004, 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024623240000000001, 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921549+0000", 2024-09-04T21:54:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921549+0000", 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921551+0000", 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.624e-06 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921557+0000", 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9479999999999996e-06 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921573+0000", 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6256000000000002e-05 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921581+0000", 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1419999999999993e-06 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921642+0000", 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0676e-05 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.921707+0000", 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4689000000000004e-05 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.923936+0000", 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022292269999999999 2024-09-04T21:54:02.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.923967+0000", 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0623999999999997e-05 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924012+0000", 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5138e-05 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1052 3.2 3:46fbfeaa:::rbd_header.11882e693cd0:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.924126+0000", 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.98700387899999997, 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022881680000000001, 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1052 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.955 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924126+0000", 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924126+0000", 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924128+0000", 2024-09-04T21:54:02.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4959999999999999e-06 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924134+0000", 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2899999999999999e-06 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924157+0000", 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3235999999999999e-05 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924165+0000", 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1519999999999992e-06 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924233+0000", 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7129000000000001e-05 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924330+0000", 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7546000000000004e-05 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.924565+0000", 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023511499999999999 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.925190+0000", 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062432600000000003 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.926349+0000", 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011589879999999999 2024-09-04T21:54:02.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.926388+0000", 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9085999999999998e-05 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.926414+0000", 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6804e-05 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1065 3.2 3:46fbfeaa:::rbd_header.11882e693cd0:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.374006+0000", 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.53712400000000005, 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040256550000000004, 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1065 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374006+0000", 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374006+0000", 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374010+0000", 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4699999999999998e-06 2024-09-04T21:54:02.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374025+0000", 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5872999999999999e-05 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374061+0000", 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5703e-05 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374086+0000", 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4403e-05 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374267+0000", 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018134800000000001 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.374435+0000", 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016784300000000001 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.375023+0000", 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058822099999999999 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.376593+0000", 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015696900000000001 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.377845+0000", 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001252579 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.377925+0000", 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9802999999999995e-05 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.378032+0000", 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106722 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1068 3.8 3:121ca1cb:::rbd_object_map.11882e693cd0:head [call lock.assert_locked in=17b,call] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.380776+0000", 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.53035430900000002, 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002926175, 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1068 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380776+0000", 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380776+0000", 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380779+0000", 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2140000000000001e-06 2024-09-04T21:54:02.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380794+0000", 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4790000000000001e-05 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380830+0000", 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6350000000000003e-05 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380852+0000", 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1789000000000001e-05 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.380941+0000", 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9272999999999997e-05 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.381042+0000", 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010086000000000001 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.381306+0000", 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026434400000000002 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.382872+0000", 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015655059999999999 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.383598+0000", 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072637699999999995 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.383657+0000", 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9018000000000003e-05 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.383702+0000", 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4654000000000003e-05 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.972 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1079 3.2 3:46fbfeaa:::rbd_header.11882e693cd0:head [call rbd.set_protection_status in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.391198+0000", 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.51993173100000001, 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024645520000000001, 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1079 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391198+0000", 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391198+0000", 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391200+0000", 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.457e-06 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391211+0000", 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1542999999999999e-05 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391231+0000", 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9403e-05 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391241+0000", 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0117e-05 2024-09-04T21:54:02.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391280+0000", 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9544e-05 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391323+0000", 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670999999999999e-05 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:02.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.391457+0000", 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013347200000000001 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.392312+0000", 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085589399999999995 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.393413+0000", 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001100356 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.393557+0000", 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000144485 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.393663+0000", 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010561 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1100 3.2 3:4550e1e5:::rbd_id.image_client.0-clone1:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.978 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.419074+0000", 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.49205587200000001, 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023288190000000002, 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1100 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419074+0000", 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419074+0000", 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419078+0000", 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.506e-06 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419093+0000", 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4786999999999999e-05 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419129+0000", 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6513000000000001e-05 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419145+0000", 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6056e-05 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419266+0000", 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121042 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419367+0000", 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010103999999999999 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.419495+0000", 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012753600000000001 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000835625 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.421346+0000", 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010152710000000001 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.421376+0000", 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0524000000000001e-05 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.421403+0000", 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6919000000000001e-05 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1133 3.8 3:121ca1cb:::rbd_object_map.11882e693cd0:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.498264+0000", 2024-09-04T21:54:02.984 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.41286563999999998, 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022702690000000001, 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1133 2024-09-04T21:54:02.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498264+0000", 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498264+0000", 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498268+0000", 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0410000000000002e-06 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498280+0000", 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.203e-05 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498303+0000", 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3699000000000001e-05 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498314+0000", 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0268000000000001e-05 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498445+0000", 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013124100000000001 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498641+0000", 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019666799999999999 2024-09-04T21:54:02.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.498966+0000", 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032488100000000001 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.500115+0000", 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001148303 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.500448+0000", 2024-09-04T21:54:02.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00033341099999999998 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.500494+0000", 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6100000000000002e-05 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.500535+0000", 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0627000000000003e-05 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.990 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1136 3.2 3:46fbfeaa:::rbd_header.11882e693cd0:head [watch unwatch cookie 139952113431664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.502952+0000", 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.40817799599999999, 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021566049999999998, 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.991 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1136 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.502952+0000", 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.502952+0000", 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.502953+0000", 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9699999999999996e-07 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.502956+0000", 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9840000000000001e-06 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.502962+0000", 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0920000000000003e-06 2024-09-04T21:54:02.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.502967+0000", 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7019999999999998e-06 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503188+0000", 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000220146 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503327+0000", 2024-09-04T21:54:02.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138942 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503583+0000", 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025609800000000001 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.504474+0000", 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00089109499999999995 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.504999+0000", 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00052503999999999995 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.505037+0000", 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8107e-05 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.505109+0000", 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1802000000000001e-05 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:02.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1137 3.2 3:46fbfeaa:::rbd_header.11882e693cd0:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.503244+0000", 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.40788564900000002, 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024046889999999998, 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1137 2024-09-04T21:54:02.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503244+0000", 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503244+0000", 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503246+0000", 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.288e-06 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503251+0000", 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5160000000000002e-06 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503262+0000", 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0705e-05 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.503269+0000", 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1940000000000001e-06 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.504047+0000", 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00077736899999999998 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.504128+0000", 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1141000000000001e-05 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.505210+0000", 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00108253 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.505415+0000", 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020438100000000001 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.505649+0000", 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234565 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1202 3.18 3:1cacc5b5:::rbd_data.1188e7c968d9.00000000000000fc:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.624616+0000", 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.28651385499999998, 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021566630000000001, 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1202 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624616+0000", 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624616+0000", 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624620+0000", 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9029999999999997e-06 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624634+0000", 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3805000000000001e-05 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624680+0000", 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5691e-05 2024-09-04T21:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624714+0000", 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4320999999999999e-05 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.624887+0000", 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000173153 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.625029+0000", 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000141681 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.625268+0000", 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023901200000000001 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.626461+0000", 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001193096 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.626713+0000", 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025252199999999998 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.626745+0000", 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1443999999999999e-05 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.626773+0000", 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8034999999999998e-05 2024-09-04T21:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1233 3.18 3:1fdf08e9:::rbd_data.1188e7c968d9.0000000000000102:head [set-alloc-hint object_size 524288 write_size 524288,write 458752~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.657814+0000", 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.25331614400000002, 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0044979570000000003, 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1233 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.657814+0000", 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.657814+0000", 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.657817+0000", 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.19e-06 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.657860+0000", 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2938000000000001e-05 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.657885+0000", 2024-09-04T21:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4749999999999999e-05 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.657900+0000", 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5356e-05 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658012+0000", 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111455 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658245+0000", 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023371000000000001 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658460+0000", 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000214816 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.662184+0000", 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0037235039999999999 2024-09-04T21:54:03.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.662215+0000", 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1090000000000002e-05 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:03.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.662264+0000", 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8844999999999997e-05 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.662312+0000", 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8303000000000002e-05 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1234 3.18 3:1d455ce8:::rbd_data.1188e7c968d9.0000000000000103:head [set-alloc-hint object_size 524288 write_size 524288,write 458752~20629] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.017 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.658480+0000", 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.25265053199999998, 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0066790499999999997, 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1234 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:03.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658480+0000", 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658480+0000", 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658481+0000", 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.243e-06 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658519+0000", 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7892000000000002e-05 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658536+0000", 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7430000000000001e-05 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658546+0000", 2024-09-04T21:54:03.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.594e-06 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.659059+0000", 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00051368900000000001 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.659205+0000", 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014517800000000001 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.659353+0000", 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014863499999999999 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665063+0000", 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057098779999999998 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665084+0000", 2024-09-04T21:54:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1290000000000001e-05 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665125+0000", 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0876e-05 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665159+0000", 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3345e-05 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1249 3.b 3:d7b078b8:::rbd_data.1188e7c968d9.0000000000000088:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.678920+0000", 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.232209944, 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021367109999999999, 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1249 2024-09-04T21:54:03.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.678920+0000", 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.678920+0000", 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.678922+0000", 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1229999999999998e-06 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.678935+0000", 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2472e-05 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.678955+0000", 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9885e-05 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.678965+0000", 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0631e-05 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.679035+0000", 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9624999999999998e-05 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.679156+0000", 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121002 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.679534+0000", 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00037795100000000001 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.680975+0000", 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001440991 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.680987+0000", 2024-09-04T21:54:03.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2638e-05 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.681022+0000", 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4888000000000002e-05 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.681057+0000", 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4505000000000003e-05 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1277 3.b 3:d3ac7ddb:::rbd_data.1188e7c968d9.000000000000018a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.758204+0000", 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.15292630900000001, 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029963030000000001, 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1277 2024-09-04T21:54:03.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758204+0000", 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758204+0000", 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:03.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758208+0000", 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5700000000000003e-06 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758220+0000", 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1897e-05 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758256+0000", 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5571999999999998e-05 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758273+0000", 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7030000000000001e-05 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758406+0000", 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013278500000000001 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758546+0000", 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013990500000000001 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.758711+0000", 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016525000000000001 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.759671+0000", 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00095969600000000001 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.761095+0000", 2024-09-04T21:54:03.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001424123 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.761152+0000", 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7405999999999999e-05 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.761200+0000", 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8068999999999997e-05 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:03.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:03.036 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:03.037 INFO:teuthology.orchestra.run.smithi112.stdout:564 write 0xe16f4fa thru 0xe16ff23 (0xa2a bytes) 2024-09-04T21:54:03.037 INFO:teuthology.orchestra.run.smithi112.stdout:566 read 0x48dcaae thru 0x48dd092 (0x5e5 bytes) 2024-09-04T21:54:03.037 INFO:teuthology.orchestra.run.smithi112.stdout:567 write 0xb8f18a7 thru 0xb8f58b0 (0x400a bytes) 2024-09-04T21:54:03.037 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.037 INFO:teuthology.orchestra.run.smithi112.stdout:569 write 0x49963c6 thru 0x499d51e (0x7159 bytes) 2024-09-04T21:54:03.038 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:03.039 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.039 INFO:teuthology.orchestra.run.smithi112.stdout:571 trunc from 0xe16ff24 to 0xc7a62f 2024-09-04T21:54:03.039 INFO:teuthology.orchestra.run.smithi112.stdout:572 write 0xbff9e2a thru 0xc004de1 (0xafb8 bytes) 2024-09-04T21:54:03.040 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.040 INFO:teuthology.orchestra.run.smithi112.stdout:574 punch from 0x9927139 to 0x9936a9f, (0xf966 bytes) 2024-09-04T21:54:03.040 INFO:teuthology.orchestra.run.smithi112.stdout:575 read 0x97d5519 thru 0x97e1cbf (0xc7a7 bytes) 2024-09-04T21:54:03.040 INFO:teuthology.orchestra.run.smithi112.stdout:576 write 0x52eed7c thru 0x52f3d1e (0x4fa3 bytes) 2024-09-04T21:54:03.040 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:03.036+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:03.040 INFO:teuthology.orchestra.run.smithi112.stdout:578 read 0x84f6980 thru 0x84ff8e2 (0x8f63 bytes) 2024-09-04T21:54:03.040 INFO:teuthology.orchestra.run.smithi112.stdout:579 read 0x9c5e9d thru 0x9d0e94 (0xaff8 bytes) 2024-09-04T21:54:03.041 INFO:teuthology.orchestra.run.smithi112.stdout:581 trunc from 0xc004de2 to 0xbe62f51 2024-09-04T21:54:03.061 INFO:teuthology.orchestra.run.smithi112.stdout:582 trunc from 0xbe62f51 to 0x8808dcd 2024-09-04T21:54:03.077 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.077 INFO:teuthology.orchestra.run.smithi112.stdout:584 read 0x3afea12 thru 0x3b05226 (0x6815 bytes) 2024-09-04T21:54:03.096 INFO:teuthology.orchestra.run.smithi112.stdout:585 trunc from 0x8808dcd to 0x1a0cc3b 2024-09-04T21:54:03.112 INFO:teuthology.orchestra.run.smithi112.stdout:586 trunc from 0x1a0cc3b to 0x2113a3b 2024-09-04T21:54:03.138 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:03.136+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:03.152 INFO:teuthology.orchestra.run.smithi112.stdout:587 writesame 0x971f59c thru 0x97275c5 data_size 0x55(0x802a bytes) 2024-09-04T21:54:03.152 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.167 INFO:teuthology.orchestra.run.smithi112.stdout:590 write 0xbda020c thru 0xbda7d28 (0x7b1d bytes) 2024-09-04T21:54:03.169 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:03.169 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:03.169 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:03.169 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:03.175 INFO:teuthology.orchestra.run.smithi112.stdout:591 punch from 0x49713ac to 0x4977ecc, (0x6b20 bytes) 2024-09-04T21:54:03.175 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.175 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.175 INFO:teuthology.orchestra.run.smithi112.stdout:595 writesame 0x658a822 thru 0x6594793 data_size 0x3(0x9f72 bytes) 2024-09-04T21:54:03.185 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:03.189 INFO:teuthology.orchestra.run.smithi112.stdout:598 writesame 0x3fa9e6 thru 0x3fe68b data_size 0x7(0x3ca6 bytes) 2024-09-04T21:54:03.202 INFO:teuthology.orchestra.run.smithi112.stdout:599 read 0x31efc5b thru 0x31f0bd2 (0xf78 bytes) 2024-09-04T21:54:03.203 INFO:teuthology.orchestra.run.smithi112.stdout:600 write 0x97a5b47 thru 0x97a8e78 (0x3332 bytes) 2024-09-04T21:54:03.218 INFO:teuthology.orchestra.run.smithi112.stdout:601 writesame 0xc536883 thru 0xc54475c data_size 0x1(0xdeda bytes) 2024-09-04T21:54:03.218 INFO:teuthology.orchestra.run.smithi112.stdout:602 read 0xa96cfd4 thru 0xa974948 (0x7975 bytes) 2024-09-04T21:54:03.218 INFO:teuthology.orchestra.run.smithi112.stdout:603 write 0x6e9ccdc thru 0x6ea4080 (0x73a5 bytes) 2024-09-04T21:54:03.226 INFO:teuthology.orchestra.run.smithi112.stdout:604 punch from 0xb7bd9eb to 0xb7cbe0a, (0xe41f bytes) 2024-09-04T21:54:03.239 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:03.235+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:03.246 INFO:teuthology.orchestra.run.smithi112.stdout:605 trunc from 0xc54475d to 0x4c87406 2024-09-04T21:54:03.266 INFO:teuthology.orchestra.run.smithi112.stdout:606 punch from 0x405300e to 0x40601d5, (0xd1c7 bytes) 2024-09-04T21:54:03.290 INFO:teuthology.orchestra.run.smithi112.stdout:607 trunc from 0x4c87406 to 0xdb1f14b 2024-09-04T21:54:03.300 INFO:teuthology.orchestra.run.smithi112.stdout:609 write 0xef8040 thru 0xefdefb (0x5ebc bytes) 2024-09-04T21:54:03.306 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.313 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:03.313 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:03.313 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:03.314 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:03.314 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:03.318 INFO:teuthology.orchestra.run.smithi112.stdout:612 write 0xed5a171 thru 0xed60b93 (0x6a23 bytes) 2024-09-04T21:54:03.325 INFO:teuthology.orchestra.run.smithi112.stdout:613 write 0x55d25e7 thru 0x55e1317 (0xed31 bytes) 2024-09-04T21:54:03.330 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:03.332 INFO:teuthology.orchestra.run.smithi112.stdout:618 read 0x1757915 thru 0x1759dfa (0x24e6 bytes) 2024-09-04T21:54:03.332 INFO:teuthology.orchestra.run.smithi112.stdout:619 write 0xd5c4430 thru 0xd5d08bd (0xc48e bytes) 2024-09-04T21:54:03.339 INFO:teuthology.orchestra.run.smithi112.stdout:620 read 0x25f99fd thru 0x2602b62 (0x9166 bytes) 2024-09-04T21:54:03.339 INFO:teuthology.orchestra.run.smithi112.stdout:621 write 0xd40837e thru 0xd41588f (0xd512 bytes) 2024-09-04T21:54:03.340 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:03.340+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:03.352 INFO:teuthology.orchestra.run.smithi112.stdout:622 trunc from 0xed60b94 to 0xcea7821 2024-09-04T21:54:03.381 INFO:teuthology.orchestra.run.smithi112.stdout:623 write 0xeb78251 thru 0xeb78406 (0x1b6 bytes) 2024-09-04T21:54:03.388 INFO:teuthology.orchestra.run.smithi112.stdout:627 read 0xc25dd99 thru 0xc269723 (0xb98b bytes) 2024-09-04T21:54:03.388 INFO:teuthology.orchestra.run.smithi112.stdout:628 read 0x3927715 thru 0x3932862 (0xb14e bytes) 2024-09-04T21:54:03.388 INFO:teuthology.orchestra.run.smithi112.stdout:629 writesame 0xc6f14e0 thru 0xc6f1fc2 data_size 0x1(0xae3 bytes) 2024-09-04T21:54:03.392 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:54:03.395 INFO:teuthology.orchestra.run.smithi112.stdout:631 read 0x7675f9a thru 0x767fb84 (0x9beb bytes) 2024-09-04T21:54:03.395 INFO:teuthology.orchestra.run.smithi112.stdout:632 write 0xb7178b7 thru 0xb7258f1 (0xe03b bytes) 2024-09-04T21:54:03.401 INFO:teuthology.orchestra.run.smithi112.stdout:634 writesame 0x86d3f6d thru 0x86d8233 data_size 0x5(0x42c7 bytes) 2024-09-04T21:54:03.415 INFO:teuthology.orchestra.run.smithi112.stdout:635 trunc from 0xeb78407 to 0xc478ee4 2024-09-04T21:54:03.429 INFO:teuthology.orchestra.run.smithi112.stdout:636 write 0x3d86daf thru 0x3d892e5 (0x2537 bytes) 2024-09-04T21:54:03.440 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:03.435+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:03.446 INFO:teuthology.orchestra.run.smithi112.stdout:637 write 0xe8bca7a thru 0xe8c27c8 (0x5d4f bytes) 2024-09-04T21:54:03.452 INFO:teuthology.orchestra.run.smithi112.stdout:639 read 0xb84077e thru 0xb84a98a (0xa20d bytes) 2024-09-04T21:54:03.452 INFO:teuthology.orchestra.run.smithi112.stdout:641 read 0xe0b08b thru 0xe0f23a (0x41b0 bytes) 2024-09-04T21:54:03.452 INFO:teuthology.orchestra.run.smithi112.stdout:642 write 0x80476 thru 0x818cd (0x1458 bytes) 2024-09-04T21:54:03.462 INFO:teuthology.orchestra.run.smithi112.stdout:643 write 0x20a256a thru 0x20a590d (0x33a4 bytes) 2024-09-04T21:54:03.467 INFO:teuthology.orchestra.run.smithi112.stdout:644 write 0x729c070 thru 0x72a03c8 (0x4359 bytes) 2024-09-04T21:54:03.472 INFO:teuthology.orchestra.run.smithi112.stdout:645 writesame 0xe2df07f thru 0xe2e7e8e data_size 0x8(0x8e10 bytes) 2024-09-04T21:54:03.478 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:648 read 0xd2a52d9 thru 0xd2aec7b (0x99a3 bytes) 2024-09-04T21:54:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:650 read 0x401df0f thru 0x401edcc (0xebe bytes) 2024-09-04T21:54:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:651 read 0x6b0c27b thru 0x6b0dd12 (0x1a98 bytes) 2024-09-04T21:54:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:653 read 0x8ecb763 thru 0x8ecc539 (0xdd7 bytes) 2024-09-04T21:54:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:654 writesame 0x6c4034b thru 0x6c4b236 data_size 0xa(0xaeec bytes) 2024-09-04T21:54:03.484 INFO:teuthology.orchestra.run.smithi112.stdout:655 write 0x5976dda thru 0x5979a3e (0x2c65 bytes) 2024-09-04T21:54:03.491 INFO:teuthology.orchestra.run.smithi112.stdout:657 trunc from 0xe8c27c9 to 0xdb50113 2024-09-04T21:54:03.504 INFO:teuthology.orchestra.run.smithi112.stdout:658 write 0xe05b3b thru 0xe0cfec (0x74b2 bytes) 2024-09-04T21:54:03.509 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.509 INFO:teuthology.orchestra.run.smithi112.stdout:660 write 0xc564ee thru 0xc5dfc2 (0x7ad5 bytes) 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:991 3.11 3:8e007a0b:::rbd_data.11882e693cd0.0000000000000031:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.824982+0000", 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.6869498549999999, 2024-09-04T21:54:03.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.008839164, 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 991 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824982+0000", 2024-09-04T21:54:03.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824982+0000", 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824984+0000", 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824993+0000", 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6109999999999994e-06 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825008+0000", 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5741999999999999e-05 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825016+0000", 2024-09-04T21:54:03.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3119999999999996e-06 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825089+0000", 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2990000000000004e-05 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825180+0000", 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1741000000000001e-05 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.833719+0000", 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0085383279999999995 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.833761+0000", 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1868e-05 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.833821+0000", 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0525000000000001e-05 2024-09-04T21:54:03.518 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1023 3.13 3:ca6dc6b8:::rbd_data.11882e693cd0.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.881317+0000", 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.6306149429999999, 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004563554, 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1023 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881317+0000", 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881317+0000", 2024-09-04T21:54:03.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881318+0000", 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.113e-06 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881323+0000", 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9049999999999996e-06 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881335+0000", 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1952e-05 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881341+0000", 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4400000000000002e-06 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881393+0000", 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1218000000000003e-05 2024-09-04T21:54:03.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881441+0000", 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8801999999999998e-05 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.885784+0000", 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043429740000000003 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.885815+0000", 2024-09-04T21:54:03.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0895999999999997e-05 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.885880+0000", 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5253999999999996e-05 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1028 3.1a 3:5e1a43f5:::rbd_data.11882e693cd0.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906748+0000", 2024-09-04T21:54:03.524 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.6051843139999999, 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034791599999999998, 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1028 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906748+0000", 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906748+0000", 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906750+0000", 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.396e-06 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906760+0000", 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8439999999999993e-06 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906791+0000", 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0735999999999998e-05 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906806+0000", 2024-09-04T21:54:03.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5174000000000001e-05 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907594+0000", 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078809400000000003 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907833+0000", 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023909499999999999 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908124+0000", 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029064500000000002 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909801+0000", 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001677886 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909975+0000", 2024-09-04T21:54:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000173785 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.910116+0000", 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014066600000000001 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.910227+0000", 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110839 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1031 3.14 3:28c51851:::rbd_data.11882e693cd0.000000000000000f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906844+0000", 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.6050876039999999, 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056239519999999998, 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.531 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1031 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906844+0000", 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906844+0000", 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906846+0000", 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.24e-06 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906853+0000", 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2289999999999997e-06 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906871+0000", 2024-09-04T21:54:03.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8726999999999999e-05 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906886+0000", 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4116e-05 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908453+0000", 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015676469999999999 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908657+0000", 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020391800000000001 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912080+0000", 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003422834 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912235+0000", 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015526599999999999 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912468+0000", 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023297500000000001 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.536 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:1026 3.1b e27/22 3:da206786:::rbd_data.11882e693cd0.0000000000000007:head v 27'7, mlcod=27'5)", 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906982+0000", 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.604949722, 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048470600000000003, 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906982+0000", 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906982+0000", 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906984+0000", 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0770000000000002e-06 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906999+0000", 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4528999999999999e-05 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907758+0000", 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075932899999999999 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907801+0000", 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2595999999999997e-05 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909170+0000", 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013696909999999999 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909226+0000", 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5195999999999998e-05 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911551+0000", 2024-09-04T21:54:03.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023256380000000001 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911829+0000", 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027800399999999999 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1033 3.13 3:cd8fd6b8:::rbd_data.11882e693cd0.0000000000000011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.541 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.907646+0000", 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.604286144, 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038967009999999998, 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1033 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907646+0000", 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907646+0000", 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907648+0000", 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2670000000000001e-06 2024-09-04T21:54:03.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907658+0000", 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7170000000000003e-06 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907688+0000", 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0620000000000002e-05 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907729+0000", 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0272999999999999e-05 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907889+0000", 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016074199999999999 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908086+0000", 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019644500000000001 2024-09-04T21:54:03.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911170+0000", 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030837199999999999 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911320+0000", 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015022500000000001 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911542+0000", 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000222692 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1034 3.f 3:f5618e06:::rbd_data.11882e693cd0.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.907776+0000", 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.604155738, 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037740719999999998, 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.547 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1034 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907776+0000", 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907776+0000", 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907778+0000", 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7239999999999999e-06 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907786+0000", 2024-09-04T21:54:03.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8429999999999993e-06 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907808+0000", 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2116e-05 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907824+0000", 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6390000000000001e-05 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908068+0000", 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000244229 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908284+0000", 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021556799999999999 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911210+0000", 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029255470000000001 2024-09-04T21:54:03.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911337+0000", 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127043 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911550+0000", 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021361199999999999 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:1027 3.1c e27/22 3:3c9626f4:::rbd_data.11882e693cd0.0000000000000008:head v 27'8, mlcod=26'6)", 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.907886+0000", 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.604045792, 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039703899999999999, 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907886+0000", 2024-09-04T21:54:03.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907886+0000", 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907887+0000", 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.314e-06 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907921+0000", 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3155999999999999e-05 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907941+0000", 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0613000000000001e-05 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907956+0000", 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4693e-05 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908963+0000", 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001007574 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909022+0000", 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8729e-05 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911319+0000", 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022969219999999999 2024-09-04T21:54:03.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911856+0000", 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053738899999999999 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:1035 3.1b e27/22 3:d8cf32f8:::rbd_data.11882e693cd0.0000000000000013:head v 27'8, mlcod=27'5)", 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.908248+0000", 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.603683408, 2024-09-04T21:54:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055671970000000003, 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908248+0000", 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908248+0000", 2024-09-04T21:54:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908250+0000", 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.784e-06 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908264+0000", 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.418e-05 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908294+0000", 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9689999999999999e-05 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908313+0000", 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8859999999999999e-05 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909829+0000", 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015164829999999999 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909897+0000", 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8002000000000004e-05 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913682+0000", 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037848320000000001 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913816+0000", 2024-09-04T21:54:03.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000133366 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1041 3.13 3:ca6dc6b8:::rbd_data.11882e693cd0.0000000000000021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.911934+0000", 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5999978960000001, 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003696436, 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.562 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1041 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911934+0000", 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911934+0000", 2024-09-04T21:54:03.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911937+0000", 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.554e-06 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911949+0000", 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2806000000000001e-05 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911975+0000", 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6040000000000001e-05 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911995+0000", 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9887000000000001e-05 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912603+0000", 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060750599999999995 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912805+0000", 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000202635 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915238+0000", 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024326539999999998 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915377+0000", 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013919900000000001 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915630+0000", 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025315500000000002 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1049 3.1a 3:5cdc6142:::rbd_data.11882e693cd0.0000000000000037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.914261+0000", 2024-09-04T21:54:03.567 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5976712210000001, 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034077840000000001, 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1049 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914261+0000", 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914261+0000", 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914272+0000", 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.094e-05 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914283+0000", 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1327e-05 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914385+0000", 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000101959 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914413+0000", 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.775e-05 2024-09-04T21:54:03.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914624+0000", 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021182199999999999 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915228+0000", 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060363300000000001 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915542+0000", 2024-09-04T21:54:03.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031385400000000002 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.916882+0000", 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013399620000000001 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917443+0000", 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00056113199999999999 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917540+0000", 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6836000000000006e-05 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917668+0000", 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012856900000000001 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.573 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1337 3.3 3:c65d6ddd:::rbd_data.1188e7c968d9.000000000000010c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.853530+0000", 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.65840163100000004, 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004973084, 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1337 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853530+0000", 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853530+0000", 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853533+0000", 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.61e-06 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853545+0000", 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1946e-05 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853594+0000", 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9095999999999999e-05 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853615+0000", 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1146999999999999e-05 2024-09-04T21:54:03.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853787+0000", 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017151199999999999 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853983+0000", 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019677900000000001 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854231+0000", 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024740100000000001 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855109+0000", 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00087801800000000005 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858414+0000", 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033049630000000002 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858469+0000", 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4981000000000001e-05 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858503+0000", 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4631000000000001e-05 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1395 3.13 3:cefc758f:::rbd_data.1188e7c968d9.0000000000000091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.579 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.962722+0000", 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.54921000399999997, 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00307997, 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1395 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962722+0000", 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962722+0000", 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962724+0000", 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2740000000000002e-06 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962732+0000", 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8699999999999992e-06 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962755+0000", 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3062999999999999e-05 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962776+0000", 2024-09-04T21:54:03.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0440000000000001e-05 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963114+0000", 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033804100000000003 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963325+0000", 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211435 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965436+0000", 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021114889999999998 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965553+0000", 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000116146 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965802+0000", 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024921200000000001 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.584 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1402 3.13 3:ce34cc0d:::rbd_data.1188e7c968d9.00000000000001c5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.962922+0000", 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.54900994800000003, 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034487609999999998, 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.585 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.586 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1402 2024-09-04T21:54:03.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.586 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962922+0000", 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962922+0000", 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962923+0000", 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.367e-06 2024-09-04T21:54:03.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962929+0000", 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962948+0000", 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8275999999999999e-05 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962962+0000", 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4453e-05 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.964441+0000", 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001479001 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.964652+0000", 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021130500000000001 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.966251+0000", 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00159885 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.966298+0000", 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7111999999999999e-05 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.966371+0000", 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2304000000000004e-05 2024-09-04T21:54:03.590 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1413 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 3702784~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.969533+0000", 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.542398881, 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030957409999999999, 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.591 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1413 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969533+0000", 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969533+0000", 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969534+0000", 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0999999999999999e-07 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969536+0000", 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.368e-06 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969543+0000", 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8719999999999996e-06 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969549+0000", 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9039999999999997e-06 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972333+0000", 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027843019999999998 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972438+0000", 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010518600000000001 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972629+0000", 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019059900000000001 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1414 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.969555+0000", 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.54237648100000002, 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003249568, 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1414 2024-09-04T21:54:03.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969555+0000", 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969555+0000", 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969556+0000", 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9700000000000002e-07 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969558+0000", 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.509e-06 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969565+0000", 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4150000000000004e-06 2024-09-04T21:54:03.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969570+0000", 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7479999999999999e-06 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972669+0000", 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003098192 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972719+0000", 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0194000000000001e-05 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972805+0000", 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6112999999999994e-05 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.600 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1415 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 4030464~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.54234713000000001, 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003816645, 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1415 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.601 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3900000000000005e-07 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969588+0000", 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6079999999999998e-06 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969595+0000", 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0820000000000001e-06 2024-09-04T21:54:03.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969601+0000", 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5389999999999998e-06 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972842+0000", 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032410770000000002 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972955+0000", 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011378 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.973401+0000", 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044601999999999999 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1520 3.3 3:c485019f:::rbd_data.1188e7c968d9.00000000000000cc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.182672+0000", 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.329260259, 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0087245150000000004, 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1520 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.606 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182672+0000", 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182672+0000", 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182673+0000", 2024-09-04T21:54:03.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4699999999999999e-06 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182679+0000", 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3929999999999996e-06 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182692+0000", 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2317000000000001e-05 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182698+0000", 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9909999999999999e-06 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182785+0000", 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7075000000000002e-05 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182999+0000", 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000214446 2024-09-04T21:54:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.184586+0000", 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015862090000000001 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191212+0000", 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066266149999999998 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191238+0000", 2024-09-04T21:54:03.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6319999999999999e-05 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191317+0000", 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8943000000000006e-05 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191396+0000", 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8736e-05 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.611 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1538 3.1a 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.198855+0000", 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.31307726200000002, 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054839449999999996, 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.612 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1538 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.198855+0000", 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.198855+0000", 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.198856+0000", 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6020000000000001e-06 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199398+0000", 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054148200000000003 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199418+0000", 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0542000000000001e-05 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199427+0000", 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4980000000000006e-06 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199489+0000", 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1908000000000004e-05 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199648+0000", 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000159841 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.200091+0000", 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000442557 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.202029+0000", 2024-09-04T21:54:03.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00193797 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.204237+0000", 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022081589999999999 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.204293+0000", 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5475e-05 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.204339+0000", 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5911000000000001e-05 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1613 3.3 3:c7196aef:::rbd_data.1188e7c968d9.00000000000001a5:head [set-alloc-hint object_size 524288 write_size 524288,write 425984~22672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.344480+0000", 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.16745201100000001, 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036000229999999999, 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:03.618 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1613 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344480+0000", 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344480+0000", 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344481+0000", 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.525e-06 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:03.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344514+0000", 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2913000000000001e-05 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344537+0000", 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2838000000000001e-05 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344546+0000", 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4449999999999996e-06 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344638+0000", 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1953999999999994e-05 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344781+0000", 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014343900000000001 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344864+0000", 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2782999999999997e-05 2024-09-04T21:54:03.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.345530+0000", 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066639100000000001 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.348026+0000", 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024954130000000001 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.348056+0000", 2024-09-04T21:54:03.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0611999999999998e-05 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.348080+0000", 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3710000000000002e-05 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:03.624 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:03.628 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:661 writesame 0x1e1dd51 thru 0x1e229aa data_size 0x1d(0x4c5a bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:665 write 0xc4d0ec1 thru 0xc4e0112 (0xf252 bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:666 read 0x416351 thru 0x417478 (0x1128 bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:667 read 0x3d40a33 thru 0x3d42315 (0x18e3 bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:669 punch from 0x1e8054e to 0x1e8bbeb, (0xb69d bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:670 write 0x6dc9166 thru 0x6dd88ef (0xf78a bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:671 read 0x449c0a2 thru 0x44a5024 (0x8f83 bytes) 2024-09-04T21:54:03.630 INFO:teuthology.orchestra.run.smithi112.stdout:672 write 0xdd01d16 thru 0xdd11128 (0xf413 bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:673 write 0xbc093ac thru 0xbc1585a (0xc4af bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:674 write 0x6473f39 thru 0x648037b (0xc443 bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:675 read 0x7c63d25 thru 0x7c6bc99 (0x7f75 bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:676 read 0x3ac3e3f thru 0x3ac596b (0x1b2d bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:678 read 0x95ba368 thru 0x95c5ed3 (0xbb6c bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:680 punch from 0x9ef21e0 to 0x9ef7116, (0x4f36 bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:681 write 0x2986876 thru 0x298bd7d (0x5508 bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:683 writesame 0xcf97254 thru 0xcf97a0a data_size 0x19(0x7b7 bytes) 2024-09-04T21:54:03.631 INFO:teuthology.orchestra.run.smithi112.stdout:684 clone 2 order 18 su 2048 sc 2 2024-09-04T21:54:03.631 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:03.624+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:03.726 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:03.724+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:03.801 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:03.801 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:03.801 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:03.801 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:03.815 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:03.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:03.862 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:03.859+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:03.928 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:03.928+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:03.937 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:03.937 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:03.937 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:03.937 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:03.937 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:03.952 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:04.029 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:04.028+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:04.056 INFO:teuthology.orchestra.run.smithi157.stdout: marking collection for removal 2024-09-04T21:54:04.056 INFO:teuthology.orchestra.run.smithi157.stdout:setting '_remove' omap key 2024-09-04T21:54:04.056 INFO:teuthology.orchestra.run.smithi157.stdout:finish_remove_pgs 2.7_head removing 2.7 2024-09-04T21:54:04.056 INFO:teuthology.orchestra.run.smithi157.stdout:Remove successful 2024-09-04T21:54:04.122 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 17.316306954000002, 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068597409999999999, 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.123 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.124 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.124 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.124 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4208", 2024-09-04T21:54:04.124 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/1349276815", 2024-09-04T21:54:04.124 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5 2024-09-04T21:54:04.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.125 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:04.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:04.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804261+0000", 2024-09-04T21:54:04.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0230000000000001e-06 2024-09-04T21:54:04.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804276+0000", 2024-09-04T21:54:04.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4304000000000001e-05 2024-09-04T21:54:04.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804309+0000", 2024-09-04T21:54:04.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3463000000000001e-05 2024-09-04T21:54:04.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804328+0000", 2024-09-04T21:54:04.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9307e-05 2024-09-04T21:54:04.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804489+0000", 2024-09-04T21:54:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000160672 2024-09-04T21:54:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804499+0000", 2024-09-04T21:54:04.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0283e-05 2024-09-04T21:54:04.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.807660+0000", 2024-09-04T21:54:04.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003160969 2024-09-04T21:54:04.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.807887+0000", 2024-09-04T21:54:04.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022673800000000001 2024-09-04T21:54:04.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:04.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.808103+0000", 2024-09-04T21:54:04.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021598900000000001 2024-09-04T21:54:04.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.809574+0000", 2024-09-04T21:54:04.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014714940000000001 2024-09-04T21:54:04.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.810928+0000", 2024-09-04T21:54:04.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001353344 2024-09-04T21:54:04.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.811009+0000", 2024-09-04T21:54:04.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1406e-05 2024-09-04T21:54:04.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.811118+0000", 2024-09-04T21:54:04.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108749 2024-09-04T21:54:04.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.139 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.140 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-04T21:54:04.140 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:04.140 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.311949648999999, 2024-09-04T21:54:04.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0075919359999999996, 2024-09-04T21:54:04.141 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.141 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.141 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.142 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.142 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4208", 2024-09-04T21:54:04.142 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/1349276815", 2024-09-04T21:54:04.142 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7 2024-09-04T21:54:04.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.142 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:04.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:04.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808619+0000", 2024-09-04T21:54:04.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:04.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808652+0000", 2024-09-04T21:54:04.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2842999999999997e-05 2024-09-04T21:54:04.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808686+0000", 2024-09-04T21:54:04.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4205999999999997e-05 2024-09-04T21:54:04.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808703+0000", 2024-09-04T21:54:04.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7597000000000001e-05 2024-09-04T21:54:04.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808872+0000", 2024-09-04T21:54:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016823599999999999 2024-09-04T21:54:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808889+0000", 2024-09-04T21:54:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7716000000000001e-05 2024-09-04T21:54:04.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813206+0000", 2024-09-04T21:54:04.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043164680000000004 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813380+0000", 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017376199999999999 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813595+0000", 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021576899999999999 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.815122+0000", 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015264339999999999 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816053+0000", 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00093072599999999999 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816141+0000", 2024-09-04T21:54:04.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8462999999999996e-05 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816207+0000", 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6429999999999999e-05 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1027 3.1c 3:3c9626f4:::rbd_data.11882e693cd0.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:04.155 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.213843738, 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056794660000000002, 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1027 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906724+0000", 2024-09-04T21:54:04.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4739999999999999e-06 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906736+0000", 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2e-05 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906768+0000", 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2106000000000001e-05 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906789+0000", 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1313000000000001e-05 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906990+0000", 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020057300000000001 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907213+0000", 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022263099999999999 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907521+0000", 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000308485 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909663+0000", 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021414559999999999 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912156+0000", 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002493033 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912289+0000", 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013296900000000001 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912401+0000", 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112426 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.161 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1040 3.1d 3:bc67932a:::rbd_data.11882e693cd0.000000000000001c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.911466+0000", 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.2090996289999998, 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044707130000000003, 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.162 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1040 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911466+0000", 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911466+0000", 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911468+0000", 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.593e-06 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911479+0000", 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0966e-05 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911518+0000", 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8757000000000002e-05 2024-09-04T21:54:04.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911539+0000", 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1216000000000002e-05 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911694+0000", 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015505299999999999 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911901+0000", 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020712500000000001 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912209+0000", 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030819799999999998 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.914546+0000", 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023369480000000002 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915765+0000", 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001218869 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915868+0000", 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010281300000000001 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915936+0000", 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8175000000000004e-05 2024-09-04T21:54:04.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1187 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578279+0000", 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.542286085, 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052155020000000003, 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1187 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578279+0000", 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578279+0000", 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578281+0000", 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4339999999999999e-06 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578286+0000", 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8540000000000002e-06 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578301+0000", 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4379000000000001e-05 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578308+0000", 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3590000000000002e-06 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9577999999999998e-05 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578459+0000", 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0947999999999997e-05 2024-09-04T21:54:04.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578813+0000", 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035448199999999999 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583085+0000", 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042723209999999999 2024-09-04T21:54:04.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583225+0000", 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013905800000000001 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583374+0000", 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014990700000000001 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583495+0000", 2024-09-04T21:54:04.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120182 2024-09-04T21:54:04.175 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.175 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.175 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1188 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5422254579999999, 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054046620000000002, 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.176 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1188 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1900000000000002e-07 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578348+0000", 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3560000000000004e-06 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578355+0000", 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.114e-06 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578359+0000", 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4660000000000001e-06 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.579323+0000", 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096376800000000005 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.579621+0000", 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029837100000000002 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.580410+0000", 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078867200000000003 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583596+0000", 2024-09-04T21:54:04.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031858160000000002 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583615+0000", 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9224000000000001e-05 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583678+0000", 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2613999999999994e-05 2024-09-04T21:54:04.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583744+0000", 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6642000000000003e-05 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1189 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.542197359, 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056526730000000004, 2024-09-04T21:54:04.182 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1189 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6100000000000002e-07 2024-09-04T21:54:04.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578370+0000", 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2479999999999999e-06 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578375+0000", 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3819999999999997e-06 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578380+0000", 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0509999999999996e-06 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.581409+0000", 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030287550000000002 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.581628+0000", 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021934499999999999 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.582249+0000", 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062055900000000002 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583761+0000", 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015120960000000001 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583845+0000", 2024-09-04T21:54:04.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4357000000000006e-05 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583945+0000", 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9550999999999996e-05 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584021+0000", 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5968000000000002e-05 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1195 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.587700+0000", 2024-09-04T21:54:04.189 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5328651020000001, 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042343989999999998, 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1195 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.190 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.587700+0000", 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.587700+0000", 2024-09-04T21:54:04.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.587701+0000", 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8999999999999995e-07 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.587707+0000", 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8440000000000003e-06 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.587720+0000", 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3645999999999999e-05 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.587730+0000", 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5179999999999993e-06 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.589871+0000", 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002141418 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.590018+0000", 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014688099999999999 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.590557+0000", 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053872200000000005 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.591593+0000", 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001035844 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.591822+0000", 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022926199999999999 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.591881+0000", 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8539000000000003e-05 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.591935+0000", 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3835000000000002e-05 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1248 3.16 3:6ce50ae3:::rbd_data.1188e7c968d9.0000000000000087:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.196 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.4430739889999999, 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013362881, 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1248 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677492+0000", 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.035e-06 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677497+0000", 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5050000000000001e-06 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677507+0000", 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8679999999999994e-06 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677514+0000", 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7049999999999999e-06 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677595+0000", 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0308000000000004e-05 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677748+0000", 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000153425 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690686+0000", 2024-09-04T21:54:04.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.01293805 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690748+0000", 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2189999999999999e-05 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690854+0000", 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105795 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.201 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1263 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.715542+0000", 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.405022714, 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043188080000000004, 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.202 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1263 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.715542+0000", 2024-09-04T21:54:04.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.715542+0000", 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.715544+0000", 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0699999999999999e-06 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.715551+0000", 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8099999999999998e-06 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.715563+0000", 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.203e-05 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.715572+0000", 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.32e-06 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.716551+0000", 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0009789550000000001 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.716847+0000", 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029613300000000002 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.717632+0000", 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078501099999999996 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.719141+0000", 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015095670000000001 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.719619+0000", 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047758000000000002 2024-09-04T21:54:04.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.719738+0000", 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119123 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.719861+0000", 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012320899999999999 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.208 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1289 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.778494+0000", 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.342071182, 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043090990000000003, 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.209 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1289 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778494+0000", 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778494+0000", 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778497+0000", 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6199999999999999e-06 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778508+0000", 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1754e-05 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778541+0000", 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.243e-05 2024-09-04T21:54:04.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778562+0000", 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1186e-05 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778687+0000", 2024-09-04T21:54:04.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000124516 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778834+0000", 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014703900000000001 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.779408+0000", 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057412900000000005 2024-09-04T21:54:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.782367+0000", 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029595849999999998 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.782495+0000", 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012754199999999999 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.782681+0000", 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018612899999999999 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.782803+0000", 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012216899999999999 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1290 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.778606+0000", 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.3419588650000001, 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004782555, 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1290 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.216 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778606+0000", 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778606+0000", 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778608+0000", 2024-09-04T21:54:04.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.663e-06 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778619+0000", 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1353e-05 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778643+0000", 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3716000000000001e-05 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.778663+0000", 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0169999999999998e-05 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.780163+0000", 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014995469999999999 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.780310+0000", 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014718199999999999 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.780898+0000", 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058839299999999999 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.782954+0000", 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020557969999999998 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.783235+0000", 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000280821 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.783321+0000", 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5766999999999994e-05 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.783389+0000", 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8145999999999994e-05 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1303 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.222 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.314612211, 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0064365610000000004, 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1303 2024-09-04T21:54:04.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805954+0000", 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.285e-06 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805960+0000", 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0460000000000002e-06 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805975+0000", 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4304999999999999e-05 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805981+0000", 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5919999999999997e-06 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806084+0000", 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103027 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806256+0000", 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017127900000000001 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806853+0000", 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00059703999999999996 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.811952+0000", 2024-09-04T21:54:04.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050998559999999998 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812130+0000", 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017728899999999999 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812269+0000", 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013945900000000001 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812390+0000", 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120383 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1304 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.314555141, 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066469160000000001, 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1304 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806011+0000", 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4000000000000001e-07 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806014+0000", 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.444e-06 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806030+0000", 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5480000000000001e-05 2024-09-04T21:54:04.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806035+0000", 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0649999999999998e-06 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.807688+0000", 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016533559999999999 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.807956+0000", 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000268182 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.809072+0000", 2024-09-04T21:54:04.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001115659 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812090+0000", 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003018097 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812415+0000", 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032458399999999999 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812546+0000", 2024-09-04T21:54:04.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013182600000000001 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812657+0000", 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011048300000000001 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.236 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.237 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1305 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.237 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:04.237 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.314502893, 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073242790000000004, 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.238 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.239 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1305 2024-09-04T21:54:04.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.239 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806063+0000", 2024-09-04T21:54:04.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4400000000000001e-07 2024-09-04T21:54:04.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806065+0000", 2024-09-04T21:54:04.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5450000000000002e-06 2024-09-04T21:54:04.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806072+0000", 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806075+0000", 2024-09-04T21:54:04.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4960000000000001e-06 2024-09-04T21:54:04.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.810223+0000", 2024-09-04T21:54:04.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041482749999999999 2024-09-04T21:54:04.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.810500+0000", 2024-09-04T21:54:04.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027680199999999998 2024-09-04T21:54:04.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.811129+0000", 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062907200000000005 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812801+0000", 2024-09-04T21:54:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001671753 2024-09-04T21:54:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813238+0000", 2024-09-04T21:54:04.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043750000000000001 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813321+0000", 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2145999999999996e-05 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813387+0000", 2024-09-04T21:54:04.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6045999999999998e-05 2024-09-04T21:54:04.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.249 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1332 3.1d 3:beff5b82:::rbd_data.1188e7c968d9.00000000000000bc:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.850543+0000", 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.270022618, 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047476840000000003, 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.250 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1332 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.251 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.850543+0000", 2024-09-04T21:54:04.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.850543+0000", 2024-09-04T21:54:04.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.850544+0000", 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.252e-06 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.850558+0000", 2024-09-04T21:54:04.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.396e-05 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.850574+0000", 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6478e-05 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.850588+0000", 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.326e-05 2024-09-04T21:54:04.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.851569+0000", 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098133300000000003 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852195+0000", 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062571899999999995 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852447+0000", 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025215799999999999 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854162+0000", 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001714929 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855197+0000", 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010357840000000001 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855252+0000", 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4784000000000003e-05 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855290+0000", 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8027e-05 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1334 3.c 3:303b0518:::rbd_data.1188e7c968d9.0000000000000101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.2679712300000001, 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056592250000000004, 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1334 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852595+0000", 2024-09-04T21:54:04.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.322e-06 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852602+0000", 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3119999999999999e-06 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852623+0000", 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1106000000000001e-05 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852636+0000", 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3692e-05 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852743+0000", 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106747 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852884+0000", 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000140755 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853074+0000", 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019004499999999999 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854595+0000", 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001521175 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858170+0000", 2024-09-04T21:54:04.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035752470000000001 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858216+0000", 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5948000000000001e-05 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858253+0000", 2024-09-04T21:54:04.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6875999999999997e-05 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1445 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.032341+0000", 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0882243789999999, 2024-09-04T21:54:04.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050597009999999998, 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1445 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032341+0000", 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032341+0000", 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032342+0000", 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.004e-06 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032347+0000", 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7759999999999997e-06 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032358+0000", 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1243000000000001e-05 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032365+0000", 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5499999999999997e-06 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.034860+0000", 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024945969999999999 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.034958+0000", 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8361000000000002e-05 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.035285+0000", 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032705899999999999 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.036215+0000", 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092982399999999999 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.037323+0000", 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001107532 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.037358+0000", 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4959999999999997e-05 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.037401+0000", 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2794999999999999e-05 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1518 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.94320064299999995, 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0049234650000000001, 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.272 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1518 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1699999999999999e-07 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177369+0000", 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.092e-06 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177375+0000", 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.959e-06 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177378+0000", 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3179999999999998e-06 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.179942+0000", 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025639339999999999 2024-09-04T21:54:04.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.180036+0000", 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4159000000000004e-05 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.180366+0000", 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032972299999999997 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.181602+0000", 2024-09-04T21:54:04.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012356209999999999 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182229+0000", 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062770600000000001 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182263+0000", 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3506000000000001e-05 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182288+0000", 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5029999999999999e-05 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1785 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.573796207, 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048312010000000002, 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:04.278 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1785 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:04.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8299999999999998e-07 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546773+0000", 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4750000000000002e-06 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:04.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546777+0000", 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4170000000000002e-06 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546780+0000", 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1650000000000002e-06 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.549526+0000", 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027455219999999998 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.549627+0000", 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100929 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.550013+0000", 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038640099999999998 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551014+0000", 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010010990000000001 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551542+0000", 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00052794799999999998 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551576+0000", 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3466000000000001e-05 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551600+0000", 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4396000000000002e-05 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.283 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.284 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.284 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:04.284 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:04.284 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:04.285 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:04.285 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:04.286 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:04.286 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:04.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:04.287 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:04.287 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:04.287 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:04.284+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:04.309 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:04.309 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:04.335 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:04.335 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:04.359 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:04.360 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:04.383 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:04.384 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:04.384 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:04.386 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:04.383+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:04.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:04.390 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:54:04.422 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:04.422 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:04.437 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:04.487 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone1 from 0xdd11129 (overlap 0xc7a62f) to 0x0 2024-09-04T21:54:04.487 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:04.488+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:04.588 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:04.583+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:04.607 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.7.7 2024-09-04T21:54:04.626 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:04.626 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:04.637 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-09-04T21:54:04.641 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:04.726 INFO:teuthology.orchestra.run.smithi112.stdout:685 write 0xa3e4950 thru 0xa3e790b (0x2fbc bytes) 2024-09-04T21:54:04.730 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:04.727+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:04.742 INFO:teuthology.orchestra.run.smithi112.stdout:687 writesame 0x4aa74c7 thru 0x4ab6592 data_size 0x42(0xf0cc bytes) 2024-09-04T21:54:04.750 INFO:teuthology.orchestra.run.smithi112.stdout:689 read 0x7b05a0c thru 0x7b11a49 (0xc03e bytes) 2024-09-04T21:54:04.750 INFO:teuthology.orchestra.run.smithi112.stdout:690 punch from 0xaa72d99 to 0xaa7e713, (0xb97a bytes) 2024-09-04T21:54:04.751 INFO:teuthology.orchestra.run.smithi112.stdout:691 write 0xb8e1b0b thru 0xb8f0d5e (0xf254 bytes) 2024-09-04T21:54:04.773 INFO:teuthology.orchestra.run.smithi112.stdout:692 write 0x2acf0 thru 0x3684d (0xbb5e bytes) 2024-09-04T21:54:04.790 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:04.788+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:04.811 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:04.811 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:04.825 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:04.825 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:04.825 INFO:teuthology.orchestra.run.smithi112.stdout:695 read 0x5932b24 thru 0x593395b (0xe38 bytes) 2024-09-04T21:54:04.825 INFO:teuthology.orchestra.run.smithi112.stdout:696 write 0xa4ce910 thru 0xa4dd62e (0xed1f bytes) 2024-09-04T21:54:04.826 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:04.832 INFO:teuthology.orchestra.run.smithi112.stdout:697 write 0xcbea2dc thru 0xcbf0061 (0x5d86 bytes) 2024-09-04T21:54:04.840 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:04.840 INFO:teuthology.orchestra.run.smithi112.stdout:699 punch from 0x1b30cf4 to 0x1b37b0e, (0x6e1a bytes) 2024-09-04T21:54:04.853 INFO:teuthology.orchestra.run.smithi112.stdout:700 trunc from 0xdd11129 to 0x848d51d 2024-09-04T21:54:04.888 INFO:teuthology.orchestra.run.smithi112.stdout:701 write 0x3b5b130 thru 0x3b6a348 (0xf219 bytes) 2024-09-04T21:54:04.890 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:04.887+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:04.894 INFO:teuthology.orchestra.run.smithi112.stdout:703 punch from 0x732eabc to 0x733a3af, (0xb8f3 bytes) 2024-09-04T21:54:04.900 INFO:teuthology.orchestra.run.smithi112.stdout:705 trunc from 0x848d51d to 0x5156c1b 2024-09-04T21:54:04.920 INFO:teuthology.orchestra.run.smithi112.stdout:708 read 0x504a2bd thru 0x504afe3 (0xd27 bytes) 2024-09-04T21:54:04.938 INFO:teuthology.orchestra.run.smithi112.stdout:709 write 0xaad6c20 thru 0xaae3b1d (0xcefe bytes) 2024-09-04T21:54:04.944 INFO:teuthology.orchestra.run.smithi112.stdout:711 trunc from 0xaae3b1e to 0xaeac726 2024-09-04T21:54:04.951 INFO:teuthology.orchestra.run.smithi112.stdout:712 punch from 0x6e63527 to 0x6e6b723, (0x81fc bytes) 2024-09-04T21:54:04.965 INFO:teuthology.orchestra.run.smithi112.stdout:713 trunc from 0xaeac726 to 0x29061e0 2024-09-04T21:54:04.967 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:04.968 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:04.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:05.009 INFO:teuthology.orchestra.run.smithi112.stdout:714 read 0x4e5b1 thru 0x54165 (0x5bb5 bytes) 2024-09-04T21:54:05.009 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:05.008+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:05.010 INFO:teuthology.orchestra.run.smithi112.stdout:715 punch from 0x7557d3 to 0x7583d1, (0x2bfe bytes) 2024-09-04T21:54:05.010 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.010 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.010 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.010 INFO:teuthology.orchestra.run.smithi112.stdout:722 read 0xf1f44e thru 0xf28185 (0x8d38 bytes) 2024-09-04T21:54:05.031 INFO:teuthology.orchestra.run.smithi112.stdout:723 write 0x8efbb41 thru 0x8f056b0 (0x9b70 bytes) 2024-09-04T21:54:05.041 INFO:teuthology.orchestra.run.smithi112.stdout:724 write 0x77aad7f thru 0x77ba634 (0xf8b6 bytes) 2024-09-04T21:54:05.049 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.049 INFO:teuthology.orchestra.run.smithi112.stdout:726 punch from 0x1dfe3a1 to 0x1e07e14, (0x9a73 bytes) 2024-09-04T21:54:05.049 INFO:teuthology.orchestra.run.smithi112.stdout:727 write 0x90143b thru 0x908d56 (0x791c bytes) 2024-09-04T21:54:05.093 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:05.087+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:05.125 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.125 INFO:teuthology.orchestra.run.smithi112.stdout:730 write 0x1c7b917 thru 0x1c869f8 (0xb0e2 bytes) 2024-09-04T21:54:05.138 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:05.139 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:05.140 INFO:teuthology.orchestra.run.smithi112.stdout:731 punch from 0x638b5b2 to 0x6390709, (0x5157 bytes) 2024-09-04T21:54:05.140 INFO:teuthology.orchestra.run.smithi112.stdout:732 read 0x3271d3 thru 0x32c943 (0x5771 bytes) 2024-09-04T21:54:05.154 INFO:teuthology.orchestra.run.smithi112.stdout:734 write 0x99b3467 thru 0x99b6fcb (0x3b65 bytes) 2024-09-04T21:54:05.156 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:05.161 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.161 INFO:teuthology.orchestra.run.smithi112.stdout:736 read 0x6c8a54c thru 0x6c937c1 (0x9276 bytes) 2024-09-04T21:54:05.161 INFO:teuthology.orchestra.run.smithi112.stdout:737 write 0x9445e05 thru 0x945340e (0xd60a bytes) 2024-09-04T21:54:05.166 INFO:teuthology.orchestra.run.smithi112.stdout:738 writesame 0x4f6d1d5 thru 0x4f7aba9 data_size 0x13(0xd9d5 bytes) 2024-09-04T21:54:05.172 INFO:teuthology.orchestra.run.smithi112.stdout:740 punch from 0x92803ed to 0x928682c, (0x643f bytes) 2024-09-04T21:54:05.185 INFO:teuthology.orchestra.run.smithi112.stdout:742 trunc from 0x99b6fcc to 0x190d06e 2024-09-04T21:54:05.193 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:05.191+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:05.229 INFO:teuthology.orchestra.run.smithi112.stdout:743 write 0x96363e3 thru 0x9645b09 (0xf727 bytes) 2024-09-04T21:54:05.235 INFO:teuthology.orchestra.run.smithi112.stdout:744 read 0x66a19e thru 0x66dc8f (0x3af2 bytes) 2024-09-04T21:54:05.239 INFO:teuthology.orchestra.run.smithi112.stdout:745 read 0x1d65b52 thru 0x1d6c052 (0x6501 bytes) 2024-09-04T21:54:05.239 INFO:teuthology.orchestra.run.smithi112.stdout:746 read 0x11a155a thru 0x11ad597 (0xc03e bytes) 2024-09-04T21:54:05.239 INFO:teuthology.orchestra.run.smithi112.stdout:747 read 0x8c05a47 thru 0x8c1427e (0xe838 bytes) 2024-09-04T21:54:05.239 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.239 INFO:teuthology.orchestra.run.smithi112.stdout:749 read 0x91cfca7 thru 0x91dc28e (0xc5e8 bytes) 2024-09-04T21:54:05.253 INFO:teuthology.orchestra.run.smithi112.stdout:750 trunc from 0x9645b0a to 0x102b5b4 2024-09-04T21:54:05.291 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:05.291 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:05.294 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:05.295+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:05.296 INFO:teuthology.orchestra.run.smithi112.stdout:751 write 0x574a2a8 thru 0x575250e (0x8267 bytes) 2024-09-04T21:54:05.303 INFO:teuthology.orchestra.run.smithi112.stdout:752 punch from 0x3a6d017 to 0x3a6eed4, (0x1ebd bytes) 2024-09-04T21:54:05.304 INFO:teuthology.orchestra.run.smithi112.stdout:754 write 0x53f1dfd thru 0x53f39eb (0x1bef bytes) 2024-09-04T21:54:05.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:05.325 INFO:teuthology.orchestra.run.smithi112.stdout:755 write 0xaab6ae2 thru 0xaab7d93 (0x12b2 bytes) 2024-09-04T21:54:05.341 INFO:teuthology.orchestra.run.smithi112.stdout:756 trunc from 0xaab7d94 to 0x39732ea 2024-09-04T21:54:05.355 INFO:teuthology.orchestra.run.smithi112.stdout:757 punch from 0xdf3d0a to 0xe0384d, (0xfb43 bytes) 2024-09-04T21:54:05.355 INFO:teuthology.orchestra.run.smithi112.stdout:758 write 0x1e52c9e thru 0x1e56b19 (0x3e7c bytes) 2024-09-04T21:54:05.359 INFO:teuthology.orchestra.run.smithi112.stdout:759 read 0x3314709 thru 0x331573c (0x1034 bytes) 2024-09-04T21:54:05.381 INFO:teuthology.orchestra.run.smithi112.stdout:760 writesame 0xcd74c1d thru 0xcd76042 data_size 0x1(0x1426 bytes) 2024-09-04T21:54:05.381 INFO:teuthology.orchestra.run.smithi112.stdout:761 read 0x835fdf1 thru 0x8363086 (0x3296 bytes) 2024-09-04T21:54:05.382 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.382 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.382 INFO:teuthology.orchestra.run.smithi112.stdout:765 write 0x510390c thru 0x510fd77 (0xc46c bytes) 2024-09-04T21:54:05.386 INFO:teuthology.orchestra.run.smithi112.stdout:766 writesame 0x58468ad thru 0x5849601 data_size 0xb(0x2d55 bytes) 2024-09-04T21:54:05.390 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.390 INFO:teuthology.orchestra.run.smithi112.stdout:768 punch from 0xa6c690f to 0xa6d55c4, (0xecb5 bytes) 2024-09-04T21:54:05.391 INFO:teuthology.orchestra.run.smithi112.stdout:769 trunc from 0xcd76043 to 0xc3fb5df 2024-09-04T21:54:05.395 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:05.396+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:05.399 INFO:teuthology.orchestra.run.smithi112.stdout:770 read 0x533430a thru 0x5334f4e (0xc45 bytes) 2024-09-04T21:54:05.399 INFO:teuthology.orchestra.run.smithi112.stdout:772 read 0x6245be6 thru 0x624d502 (0x791d bytes) 2024-09-04T21:54:05.399 INFO:teuthology.orchestra.run.smithi112.stdout:774 read 0x2ae28e7 thru 0x2aef669 (0xcd83 bytes) 2024-09-04T21:54:05.399 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.399 INFO:teuthology.orchestra.run.smithi112.stdout:776 read 0x6d9b34 thru 0x6ddb54 (0x4021 bytes) 2024-09-04T21:54:05.401 INFO:teuthology.orchestra.run.smithi112.stdout:777 write 0xa85fde5 thru 0xa86e956 (0xeb72 bytes) 2024-09-04T21:54:05.426 INFO:teuthology.orchestra.run.smithi112.stdout:778 trunc from 0xc3fb5df to 0x621af0 2024-09-04T21:54:05.465 INFO:teuthology.orchestra.run.smithi112.stdout:779 read 0x54689a thru 0x54e550 (0x7cb7 bytes) 2024-09-04T21:54:05.469 INFO:teuthology.orchestra.run.smithi112.stdout:780 read 0x4476 thru 0x7406 (0x2f91 bytes) 2024-09-04T21:54:05.477 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:05.477 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:05.480 INFO:teuthology.orchestra.run.smithi112.stdout:781 trunc from 0x621af0 to 0x6ec2a51 2024-09-04T21:54:05.488 INFO:teuthology.orchestra.run.smithi112.stdout:782 read 0x3fab7c7 thru 0x3fb85ca (0xce04 bytes) 2024-09-04T21:54:05.488 INFO:teuthology.orchestra.run.smithi112.stdout:783 writesame 0x5a96323 thru 0x5a99556 data_size 0x36(0x3234 bytes) 2024-09-04T21:54:05.496 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:05.491+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:05.497 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:05.498 INFO:teuthology.orchestra.run.smithi112.stdout:784 trunc from 0x6ec2a51 to 0x8b92f04 2024-09-04T21:54:05.506 INFO:teuthology.orchestra.run.smithi112.stdout:785 writesame 0x3151d1d thru 0x3160cda data_size 0x35(0xefbe bytes) 2024-09-04T21:54:05.513 INFO:teuthology.orchestra.run.smithi112.stdout:786 write 0x18b615a thru 0x18ba681 (0x4528 bytes) 2024-09-04T21:54:05.520 INFO:teuthology.orchestra.run.smithi112.stdout:787 write 0x2da13bd thru 0x2da2c97 (0x18db bytes) 2024-09-04T21:54:05.525 INFO:teuthology.orchestra.run.smithi112.stdout:789 writesame 0x5ad97d2 thru 0x5adedda data_size 0x5(0x5609 bytes) 2024-09-04T21:54:05.529 INFO:teuthology.orchestra.run.smithi112.stdout:790 write 0x708ee6b thru 0x7099696 (0xa82c bytes) 2024-09-04T21:54:05.535 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.535 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.547 INFO:teuthology.orchestra.run.smithi112.stdout:795 write 0xbe193ec thru 0xbe263bc (0xcfd1 bytes) 2024-09-04T21:54:05.555 INFO:teuthology.orchestra.run.smithi112.stdout:796 write 0x7c59944 thru 0x7c6370b (0x9dc8 bytes) 2024-09-04T21:54:05.560 INFO:teuthology.orchestra.run.smithi112.stdout:797 writesame 0xac47993 thru 0xac4d2bd data_size 0x7(0x592b bytes) 2024-09-04T21:54:05.567 INFO:teuthology.orchestra.run.smithi112.stdout:798 write 0x1f759d1 thru 0x1f80d0e (0xb33e bytes) 2024-09-04T21:54:05.577 INFO:teuthology.orchestra.run.smithi112.stdout:799 write 0x82e7ca2 thru 0x82efda7 (0x8106 bytes) 2024-09-04T21:54:05.583 INFO:teuthology.orchestra.run.smithi112.stdout:800 read 0x59573c7 thru 0x59610cb (0x9d05 bytes) 2024-09-04T21:54:05.583 INFO:teuthology.orchestra.run.smithi112.stdout:801 writesame 0xa67108a thru 0xa679937 data_size 0x5(0x88ae bytes) 2024-09-04T21:54:05.590 INFO:teuthology.orchestra.run.smithi112.stdout:803 read 0x409f52e thru 0x40ac635 (0xd108 bytes) 2024-09-04T21:54:05.590 INFO:teuthology.orchestra.run.smithi112.stdout:804 writesame 0x837f305 thru 0x838d8d4 data_size 0x8(0xe5d0 bytes) 2024-09-04T21:54:05.597 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:05.596+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:05.598 INFO:teuthology.orchestra.run.smithi112.stdout:805 write 0x8ab3dc3 thru 0x8ab7379 (0x35b7 bytes) 2024-09-04T21:54:05.602 INFO:teuthology.orchestra.run.smithi112.stdout:807 read 0x277ae89 thru 0x27804f3 (0x566b bytes) 2024-09-04T21:54:05.602 INFO:teuthology.orchestra.run.smithi112.stdout:808 read 0x4c37735 thru 0x4c37a22 (0x2ee bytes) 2024-09-04T21:54:05.602 INFO:teuthology.orchestra.run.smithi112.stdout:809 write 0x34378b0 thru 0x343ee58 (0x75a9 bytes) 2024-09-04T21:54:05.606 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.607 INFO:teuthology.orchestra.run.smithi112.stdout:812 writesame 0x207062a thru 0x207db19 data_size 0x8(0xd4f0 bytes) 2024-09-04T21:54:05.612 INFO:teuthology.orchestra.run.smithi112.stdout:813 punch from 0x4007222 to 0x4007808, (0x5e6 bytes) 2024-09-04T21:54:05.612 INFO:teuthology.orchestra.run.smithi112.stdout:814 read 0x1f17a0b thru 0x1f26460 (0xea56 bytes) 2024-09-04T21:54:05.613 INFO:teuthology.orchestra.run.smithi112.stdout:815 read 0x559c8e7 thru 0x559ff36 (0x3650 bytes) 2024-09-04T21:54:05.613 INFO:teuthology.orchestra.run.smithi112.stdout:816 write 0x7bbad5c thru 0x7bbda5b (0x2d00 bytes) 2024-09-04T21:54:05.619 INFO:teuthology.orchestra.run.smithi112.stdout:819 writesame 0x157bdf8 thru 0x157fcc7 data_size 0x50(0x3ed0 bytes) 2024-09-04T21:54:05.626 INFO:teuthology.orchestra.run.smithi112.stdout:820 read 0x16c61d8 thru 0x16d1594 (0xb3bd bytes) 2024-09-04T21:54:05.626 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.626 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.626 INFO:teuthology.orchestra.run.smithi112.stdout:825 write 0x790520e thru 0x791402f (0xee22 bytes) 2024-09-04T21:54:05.630 INFO:teuthology.orchestra.run.smithi112.stdout:826 write 0x8f69da0 thru 0x8f74bba (0xae1b bytes) 2024-09-04T21:54:05.634 INFO:teuthology.orchestra.run.smithi112.stdout:827 read 0x7ed1223 thru 0x7ee0077 (0xee55 bytes) 2024-09-04T21:54:05.634 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.634 INFO:teuthology.orchestra.run.smithi112.stdout:831 read 0x71a8617 thru 0x71b5306 (0xccf0 bytes) 2024-09-04T21:54:05.635 INFO:teuthology.orchestra.run.smithi112.stdout:832 read 0x3a973d8 thru 0x3a9ffdf (0x8c08 bytes) 2024-09-04T21:54:05.642 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:05.642 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:05.643 INFO:teuthology.orchestra.run.smithi112.stdout:833 trunc from 0xbe263bd to 0x5e5e714 2024-09-04T21:54:05.661 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:05.661 INFO:teuthology.orchestra.run.smithi112.stdout:834 write 0x3711488 thru 0x37172c8 (0x5e41 bytes) 2024-09-04T21:54:05.668 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:05.669 INFO:teuthology.orchestra.run.smithi112.stdout:837 read 0x1f6f31f thru 0x1f79769 (0xa44b bytes) 2024-09-04T21:54:05.669 INFO:teuthology.orchestra.run.smithi112.stdout:838 write 0x3438b24 thru 0x343fd88 (0x7265 bytes) 2024-09-04T21:54:05.689 INFO:teuthology.orchestra.run.smithi112.stdout:840 write 0xdb1bc5b thru 0xdb1ca21 (0xdc7 bytes) 2024-09-04T21:54:05.697 INFO:teuthology.orchestra.run.smithi112.stdout:841 writesame 0x76c7fea thru 0x76cf20c data_size 0x1(0x7223 bytes) 2024-09-04T21:54:05.697 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:05.695+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:05.739 INFO:teuthology.orchestra.run.smithi112.stdout:842 trunc from 0xdb1ca22 to 0x228d6bc 2024-09-04T21:54:05.763 INFO:teuthology.orchestra.run.smithi112.stdout:843 writesame 0x42901f thru 0x42fcc6 data_size 0x26(0x6ca8 bytes) 2024-09-04T21:54:05.798 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:05.796+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:05.802 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:05.802 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:05.818 INFO:teuthology.orchestra.run.smithi112.stdout:844 trunc from 0x228d6bc to 0x6b18f35 2024-09-04T21:54:05.819 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:05.840 INFO:teuthology.orchestra.run.smithi112.stdout:845 writesame 0xa372877 thru 0xa37b8ae data_size 0x14(0x9038 bytes) 2024-09-04T21:54:05.852 INFO:teuthology.orchestra.run.smithi112.stdout:846 writesame 0xc78e187 thru 0xc795a37 data_size 0x3(0x78b1 bytes) 2024-09-04T21:54:05.852 INFO:teuthology.orchestra.run.smithi112.stdout:847 read 0x3eb9c82 thru 0x3ebf60b (0x598a bytes) 2024-09-04T21:54:05.852 INFO:teuthology.orchestra.run.smithi112.stdout:849 read 0x333c118 thru 0x333f8aa (0x3793 bytes) 2024-09-04T21:54:05.853 INFO:teuthology.orchestra.run.smithi112.stdout:850 trunc from 0xc795a38 to 0xd31bd50 2024-09-04T21:54:05.861 INFO:teuthology.orchestra.run.smithi112.stdout:851 punch from 0x5130394 to 0x5138819, (0x8485 bytes) 2024-09-04T21:54:05.861 INFO:teuthology.orchestra.run.smithi112.stdout:852 write 0x4284dea thru 0x4287219 (0x2430 bytes) 2024-09-04T21:54:05.866 INFO:teuthology.orchestra.run.smithi112.stdout:853 write 0x2cb7cf7 thru 0x2cc5958 (0xdc62 bytes) 2024-09-04T21:54:05.872 INFO:teuthology.orchestra.run.smithi112.stdout:854 write 0x630b199 thru 0x6310e4a (0x5cb2 bytes) 2024-09-04T21:54:05.878 INFO:teuthology.orchestra.run.smithi112.stdout:855 write 0xbec14f9 thru 0xbecb7dc (0xa2e4 bytes) 2024-09-04T21:54:05.899 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:05.896+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:05.901 INFO:teuthology.orchestra.run.smithi112.stdout:856 trunc from 0xd31bd50 to 0x33221b0 2024-09-04T21:54:05.918 INFO:teuthology.orchestra.run.smithi112.stdout:857 clone 3 order 21 su 131072 sc 7 2024-09-04T21:54:05.968 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:05.969 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:05.982 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:06.023 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:06.024+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:06.101 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:06.099+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:06.144 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:06.144 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:06.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:06.203 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:06.204+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:06.274 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:06.275 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:06.289 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:06.331 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:06.332+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:06.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:06.404 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:06.403+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:06.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:54:06.419 INFO:teuthology.orchestra.run.smithi157.stdout:get_pg_num_history pg_num_history pg_num_history(e21 pg_nums {1={14=1},2={18=8}} deleted_pools ) 2024-09-04T21:54:06.419 INFO:teuthology.orchestra.run.smithi157.stdout:Importing pgid 2.7 2024-09-04T21:54:06.420 INFO:teuthology.orchestra.run.smithi157.stdout:write_pg epoch 21 info 2.7( empty local-lis/les=18/19 n=0 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-09-04T21:54:06.425 INFO:teuthology.orchestra.run.smithi157.stdout:Import successful 2024-09-04T21:54:06.468 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:06.468 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:06.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:06.517 INFO:teuthology.orchestra.run.smithi112.stdout:flattening image image_client.0-clone2 2024-09-04T21:54:06.517 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:06.516+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:06.605 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:06.605 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:06.606 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:06.604+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:06.636 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:06.636 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:06.664 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:06.665 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:06.697 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:06.698 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:06.706 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:06.707+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:06.725 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:06.731 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:06.731 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:06.731 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:06.732 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:06.732 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:06.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:06.751 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:06.751 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:06.771 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:06.771 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:06.771 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:06.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:06.807 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:06.807+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:06.818 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-09-04T21:54:06.823 INFO:teuthology.orchestra.run.smithi112.stdout:858 trunc from 0x33221b0 to 0x49af8b1 2024-09-04T21:54:06.850 INFO:teuthology.orchestra.run.smithi112.stdout:859 write 0x4be5267 thru 0x4be7c5a (0x29f4 bytes) 2024-09-04T21:54:06.854 INFO:teuthology.orchestra.run.smithi112.stdout:860 read 0x1f861e2 thru 0x1f95ee3 (0xfd02 bytes) 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1044 3.8 3:11b7ccdb:::rbd_data.11882e693cd0.000000000000002e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:01.913330+0000", 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.9628210179999996, 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053603039999999998, 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1044 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913330+0000", 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913330+0000", 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913331+0000", 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.091e-06 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913341+0000", 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0212e-05 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913353+0000", 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1155000000000001e-05 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913360+0000", 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6580000000000002e-06 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913443+0000", 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2288999999999998e-05 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913567+0000", 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000123943 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.913783+0000", 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021682 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.915863+0000", 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020799 2024-09-04T21:54:06.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.918587+0000", 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027240400000000001 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.918644+0000", 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6436e-05 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:01.918690+0000", 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.676e-05 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1234 3.18 3:1d455ce8:::rbd_data.1188e7c968d9.0000000000000103:head [set-alloc-hint object_size 524288 write_size 524288,write 458752~20629] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.658480+0000", 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.2176716069999998, 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0066790499999999997, 2024-09-04T21:54:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1234 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658480+0000", 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658480+0000", 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658481+0000", 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.243e-06 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658519+0000", 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7892000000000002e-05 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658536+0000", 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7430000000000001e-05 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.658546+0000", 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.594e-06 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.659059+0000", 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00051368900000000001 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.659205+0000", 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014517800000000001 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.659353+0000", 2024-09-04T21:54:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014863499999999999 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665063+0000", 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057098779999999998 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665084+0000", 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1290000000000001e-05 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665125+0000", 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0876e-05 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.665159+0000", 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3345e-05 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1335 3.18 3:1fdf08e9:::rbd_data.1188e7c968d9.0000000000000102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.854204+0000", 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.0219472290000002, 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0054969670000000002, 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1335 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854204+0000", 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854204+0000", 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854206+0000", 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6649999999999999e-06 2024-09-04T21:54:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854230+0000", 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5067e-05 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854240+0000", 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.927e-06 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854385+0000", 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014500299999999999 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854601+0000", 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000216197 2024-09-04T21:54:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854742+0000", 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014094699999999999 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856515+0000", 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017723350000000001 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859625+0000", 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031104769999999999 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859665+0000", 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9616999999999997e-05 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859701+0000", 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6269999999999997e-05 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1334 3.c e28/22 3:303b0518:::rbd_data.1188e7c968d9.0000000000000101:head v 28'16, mlcod=28'14)", 2024-09-04T21:54:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.854576+0000", 2024-09-04T21:54:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.0215750569999997, 2024-09-04T21:54:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046708390000000004, 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854576+0000", 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854576+0000", 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854577+0000", 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.062e-06 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854584+0000", 2024-09-04T21:54:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855017+0000", 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00043310100000000003 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855025+0000", 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2819999999999996e-06 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855089+0000", 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4029999999999995e-05 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855111+0000", 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1543999999999999e-05 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859192+0000", 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040811099999999998 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859247+0000", 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5207000000000002e-05 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1336 3.18 3:1d455ce8:::rbd_data.1188e7c968d9.0000000000000103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.855077+0000", 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.0210740119999997, 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047800519999999999, 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1336 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855077+0000", 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855077+0000", 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855078+0000", 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.145e-06 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855084+0000", 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3600000000000004e-06 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855097+0000", 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3123e-05 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855105+0000", 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5969999999999992e-06 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855166+0000", 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0624000000000001e-05 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855227+0000", 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0668999999999998e-05 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.855302+0000", 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4956999999999994e-05 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859220+0000", 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0039181750000000003 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859755+0000", 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053478599999999999 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859812+0000", 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7584999999999998e-05 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.859857+0000", 2024-09-04T21:54:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5031000000000003e-05 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1519 3.15 3:a8514e88:::rbd_data.1188e7c968d9.00000000000000c8:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.182380+0000", 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.6937711559999999, 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0082955660000000007, 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1519 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182380+0000", 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182380+0000", 2024-09-04T21:54:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182384+0000", 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7069999999999998e-06 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182399+0000", 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5693e-05 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182437+0000", 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7143e-05 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182456+0000", 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8957000000000002e-05 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.182626+0000", 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017021500000000001 2024-09-04T21:54:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.183013+0000", 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038734000000000002 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.190438+0000", 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074245530000000004 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.190547+0000", 2024-09-04T21:54:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010908600000000001 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.190676+0000", 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012887200000000001 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7090 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 141312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702282+0000", 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.173868827, 2024-09-04T21:54:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0048888450000000002, 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7090 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702282+0000", 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702282+0000", 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702283+0000", 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.51e-07 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702289+0000", 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702295+0000", 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6030000000000004e-06 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702300+0000", 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7910000000000004e-06 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.706710+0000", 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0044105910000000002 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.706796+0000", 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5433999999999996e-05 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707171+0000", 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00037554199999999998 2024-09-04T21:54:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7091 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 145408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702320+0000", 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17383105400000001, 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005135555, 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7091 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702320+0000", 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702320+0000", 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702321+0000", 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7599999999999997e-07 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702327+0000", 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6420000000000001e-06 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702335+0000", 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.255e-06 2024-09-04T21:54:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702340+0000", 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3170000000000001e-06 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707207+0000", 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0048666869999999997 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707296+0000", 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9189000000000003e-05 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707456+0000", 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159989 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7092 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 149504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702348+0000", 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17380319899999999, 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005460459, 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7092 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702348+0000", 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702348+0000", 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702348+0000", 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000002e-07 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702350+0000", 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0430000000000002e-06 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702357+0000", 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0270000000000001e-06 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702361+0000", 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6550000000000003e-06 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707488+0000", 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051265119999999997 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707576+0000", 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8776999999999994e-05 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707808+0000", 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000231986 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7093 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 153600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702368+0000", 2024-09-04T21:54:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17378349200000001, 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0068699369999999996, 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7093 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702368+0000", 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702368+0000", 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702368+0000", 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6000000000000004e-07 2024-09-04T21:54:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702370+0000", 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702376+0000", 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8490000000000002e-06 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702381+0000", 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7350000000000004e-06 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707842+0000", 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0054609189999999998 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.707967+0000", 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012580299999999999 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709238+0000", 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001270186 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7094 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 157696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702387+0000", 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.173764318, 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0061627619999999996, 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7094 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702387+0000", 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702387+0000", 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702387+0000", 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7099999999999997e-07 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702389+0000", 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1179999999999999e-06 2024-09-04T21:54:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702395+0000", 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8830000000000003e-06 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702400+0000", 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8450000000000002e-06 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.708271+0000", 2024-09-04T21:54:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058708750000000002 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.708358+0000", 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7264999999999999e-05 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.708550+0000", 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019140499999999999 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7095 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 161792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702406+0000", 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17374487899999999, 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065808760000000003, 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7095 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702406+0000", 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702406+0000", 2024-09-04T21:54:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702407+0000", 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702412+0000", 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1390000000000003e-06 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702418+0000", 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7989999999999999e-06 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702422+0000", 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.708579+0000", 2024-09-04T21:54:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0061564150000000002 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.708770+0000", 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191728 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.708987+0000", 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021669199999999999 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7096 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 165888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702429+0000", 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17372258199999999, 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.006917805, 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7096 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702429+0000", 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702429+0000", 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702429+0000", 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8799999999999998e-07 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702431+0000", 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7990000000000001e-06 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702437+0000", 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7769999999999999e-06 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702441+0000", 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6079999999999998e-06 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709017+0000", 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065762360000000001 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709105+0000", 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7100000000000003e-05 2024-09-04T21:54:06.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709346+0000", 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024189700000000001 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7097 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 169984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702447+0000", 2024-09-04T21:54:06.945 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.173703937, 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0072059250000000002, 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7097 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702447+0000", 2024-09-04T21:54:06.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702447+0000", 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702448+0000", 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8299999999999998e-07 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702449+0000", 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8279999999999999e-06 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702455+0000", 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6670000000000001e-06 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702464+0000", 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0129999999999999e-06 2024-09-04T21:54:06.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709414+0000", 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069494609999999997 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709500+0000", 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6116e-05 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709653+0000", 2024-09-04T21:54:06.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015345699999999999 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7098 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 174080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702472+0000", 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17367902099999999, 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0075126840000000004, 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7098 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702472+0000", 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702472+0000", 2024-09-04T21:54:06.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702473+0000", 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7e-07 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702475+0000", 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1260000000000001e-06 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702481+0000", 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4679999999999997e-06 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702486+0000", 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1869999999999996e-06 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709695+0000", 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0072088050000000004 2024-09-04T21:54:06.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709829+0000", 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013413300000000001 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.709985+0000", 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015549499999999999 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7099 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 178176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702494+0000", 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17365728599999999, 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0078726330000000004, 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7099 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.955 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702494+0000", 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702494+0000", 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702494+0000", 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702497+0000", 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2730000000000001e-06 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702503+0000", 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3289999999999999e-06 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702508+0000", 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9169999999999997e-06 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710018+0000", 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0075105049999999998 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710141+0000", 2024-09-04T21:54:06.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012291299999999999 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710367+0000", 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022526800000000001 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7100 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 182272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702515+0000", 2024-09-04T21:54:06.959 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.173636384, 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0081770780000000008, 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7100 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702515+0000", 2024-09-04T21:54:06.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702515+0000", 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702515+0000", 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5499999999999998e-07 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702517+0000", 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.035e-06 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702523+0000", 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6919999999999999e-06 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702527+0000", 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4320000000000001e-06 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710400+0000", 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0078721820000000001 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710483+0000", 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3442000000000005e-05 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710692+0000", 2024-09-04T21:54:06.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020884000000000001 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7101 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 186368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702533+0000", 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17361776900000001, 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0085050999999999998, 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.964 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7101 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702533+0000", 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702533+0000", 2024-09-04T21:54:06.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702534+0000", 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.77e-07 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702536+0000", 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8020000000000001e-06 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702541+0000", 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7620000000000001e-06 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702547+0000", 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7359999999999998e-06 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710725+0000", 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0081779000000000001 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.710811+0000", 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5876000000000002e-05 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711039+0000", 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022764700000000001 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.969 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7102 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 190464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.969 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702553+0000", 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17359797099999999, 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0087365589999999996, 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7102 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702553+0000", 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702553+0000", 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702554+0000", 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7399999999999999e-07 2024-09-04T21:54:06.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702555+0000", 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8929999999999999e-06 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702561+0000", 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6829999999999996e-06 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702566+0000", 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711073+0000", 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0085068239999999996 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711157+0000", 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4122000000000005e-05 2024-09-04T21:54:06.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711290+0000", 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132893 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7103 3.15 3:a94e422c:::rbd_data.1188e7c968d9.000000000000000d:4 [read 194560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.702593+0000", 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.17355799699999999, 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0089599640000000008, 2024-09-04T21:54:06.974 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7103 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702593+0000", 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702593+0000", 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702594+0000", 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4400000000000001e-07 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702612+0000", 2024-09-04T21:54:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8576e-05 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702619+0000", 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.702625+0000", 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2800000000000003e-06 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:06.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711316+0000", 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0086908590000000004 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711399+0000", 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3453000000000002e-05 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.711553+0000", 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015412100000000001 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.978 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:06.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:06.979 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:06.980 INFO:teuthology.orchestra.run.smithi112.stdout:862 writesame 0xd03e45d thru 0xd03ea96 data_size 0x1(0x63a bytes) 2024-09-04T21:54:06.980 INFO:teuthology.orchestra.run.smithi112.stdout:863 read 0x698a99d thru 0x698fde5 (0x5449 bytes) 2024-09-04T21:54:06.981 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /var/log/ceph/exp.2.7.7 2024-09-04T21:54:06.982 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:06.983 INFO:teuthology.orchestra.run.smithi112.stdout:865 trunc from 0xd03ea97 to 0x3a53ca9 2024-09-04T21:54:06.983 INFO:teuthology.orchestra.run.smithi112.stdout:867 read 0x2da34a1 thru 0x2da6ca8 (0x3808 bytes) 2024-09-04T21:54:06.983 INFO:teuthology.orchestra.run.smithi112.stdout:870 read 0x34490d5 thru 0x344dfde (0x4f0a bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:871 trunc from 0x3a53ca9 to 0xe6030ae 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:874 write 0x137f6e7 thru 0x138b7ea (0xc104 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:875 punch from 0x2a013f2 to 0x2a0a176, (0x8d84 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:878 punch from 0xacb2ec9 to 0xacb527e, (0x23b5 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:879 read 0xdf8abaa thru 0xdf8c5e6 (0x1a3d bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:880 read 0x2a4aba6 thru 0x2a53a5b (0x8eb6 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:881 read 0x4e8cdcb thru 0x4e9ca58 (0xfc8e bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:882 read 0xe27c582 thru 0xe27cdea (0x869 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:883 write 0x2cc62db thru 0x2cc67f1 (0x517 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:885 writesame 0xcb645 thru 0xdabb6 data_size 0x59(0xf572 bytes) 2024-09-04T21:54:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:887 punch from 0xd7e8c91 to 0xd7f5afd, (0xce6c bytes) 2024-09-04T21:54:06.985 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:06.980+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:07.002 INFO:teuthology.orchestra.run.smithi112.stdout:888 trunc from 0xe6030ae to 0x44a3abf 2024-09-04T21:54:07.030 INFO:teuthology.orchestra.run.smithi112.stdout:889 writesame 0x52a9013 thru 0x52acd1d data_size 0x1(0x3d0b bytes) 2024-09-04T21:54:07.030 INFO:teuthology.orchestra.run.smithi112.stdout:892 read 0x133043f thru 0x134028b (0xfe4d bytes) 2024-09-04T21:54:07.030 INFO:teuthology.orchestra.run.smithi112.stdout:893 read 0x5190214 thru 0x519773f (0x752c bytes) 2024-09-04T21:54:07.030 INFO:teuthology.orchestra.run.smithi112.stdout:894 punch from 0x105522c to 0x1060ed5, (0xbca9 bytes) 2024-09-04T21:54:07.030 INFO:teuthology.orchestra.run.smithi112.stdout:897 punch from 0x3726cea to 0x372d85f, (0x6b75 bytes) 2024-09-04T21:54:07.050 INFO:teuthology.orchestra.run.smithi112.stdout:899 writesame 0xa031b97 thru 0xa03feeb data_size 0x13(0xe355 bytes) 2024-09-04T21:54:07.060 INFO:teuthology.orchestra.run.smithi112.stdout:900 trunc from 0xa03feec to 0x4ed0724 2024-09-04T21:54:07.082 INFO:teuthology.orchestra.run.smithi112.stdout:901 writesame 0x603cbd4 thru 0x60489cc data_size 0x57(0xbdf9 bytes) 2024-09-04T21:54:07.082 INFO:teuthology.orchestra.run.smithi112.stdout:904 read 0x10c174c thru 0x10c4364 (0x2c19 bytes) 2024-09-04T21:54:07.086 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:07.087+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:07.093 INFO:teuthology.orchestra.run.smithi112.stdout:905 trunc from 0x60489cd to 0x9d01397 2024-09-04T21:54:07.105 INFO:teuthology.orchestra.run.smithi112.stdout:907 punch from 0x18cbf67 to 0x18d7e06, (0xbe9f bytes) 2024-09-04T21:54:07.105 INFO:teuthology.orchestra.run.smithi112.stdout:908 write 0x6c7566d thru 0x6c84d05 (0xf699 bytes) 2024-09-04T21:54:07.113 INFO:teuthology.orchestra.run.smithi112.stdout:909 read 0x3891b8a thru 0x3896d34 (0x51ab bytes) 2024-09-04T21:54:07.113 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.113 INFO:teuthology.orchestra.run.smithi112.stdout:911 writesame 0x1985db4 thru 0x198ca3e data_size 0x1(0x6c8b bytes) 2024-09-04T21:54:07.150 INFO:teuthology.orchestra.run.smithi112.stdout:913 read 0x27bba71 thru 0x27ca46a (0xe9fa bytes) 2024-09-04T21:54:07.150 INFO:teuthology.orchestra.run.smithi112.stdout:914 punch from 0x23f7d4a to 0x23fe306, (0x65bc bytes) 2024-09-04T21:54:07.157 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:07.157 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:07.157 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:07.157 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:07.166 INFO:teuthology.orchestra.run.smithi112.stdout:916 write 0xacc34d9 thru 0xaccf54d (0xc075 bytes) 2024-09-04T21:54:07.172 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:07.173 INFO:teuthology.orchestra.run.smithi112.stdout:918 write 0x819a8b9 thru 0x81a25c2 (0x7d0a bytes) 2024-09-04T21:54:07.181 INFO:teuthology.orchestra.run.smithi112.stdout:919 read 0x18a5414 thru 0x18a9273 (0x3e60 bytes) 2024-09-04T21:54:07.182 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:07.180+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:07.182 INFO:teuthology.orchestra.run.smithi112.stdout:920 write 0x9025b89 thru 0x902c947 (0x6dbf bytes) 2024-09-04T21:54:07.190 INFO:teuthology.orchestra.run.smithi112.stdout:921 write 0x11b6c64 thru 0x11c1e7c (0xb219 bytes) 2024-09-04T21:54:07.194 INFO:teuthology.orchestra.run.smithi112.stdout:922 read 0x5851724 thru 0x585e8da (0xd1b7 bytes) 2024-09-04T21:54:07.203 INFO:teuthology.orchestra.run.smithi112.stdout:923 trunc from 0xaccf54e to 0x7b38ab7 2024-09-04T21:54:07.224 INFO:teuthology.orchestra.run.smithi112.stdout:924 punch from 0x3b62e6b to 0x3b6d6dc, (0xa871 bytes) 2024-09-04T21:54:07.232 INFO:teuthology.orchestra.run.smithi112.stdout:925 trunc from 0x7b38ab7 to 0x4a6a235 2024-09-04T21:54:07.252 INFO:teuthology.orchestra.run.smithi112.stdout:926 read 0x298a8be thru 0x2998495 (0xdbd8 bytes) 2024-09-04T21:54:07.283 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:07.283+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:07.294 INFO:teuthology.orchestra.run.smithi112.stdout:927 write 0xed0bf94 thru 0xed0d6ee (0x175b bytes) 2024-09-04T21:54:07.301 INFO:teuthology.orchestra.run.smithi112.stdout:928 write 0x6df4b73 thru 0x6df716c (0x25fa bytes) 2024-09-04T21:54:07.308 INFO:teuthology.orchestra.run.smithi112.stdout:930 write 0xcd3bac6 thru 0xcd3d59d (0x1ad8 bytes) 2024-09-04T21:54:07.329 INFO:teuthology.orchestra.run.smithi112.stdout:931 trunc from 0xed0d6ef to 0x9e6d836 2024-09-04T21:54:07.333 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:07.333 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:07.333 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:07.333 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:07.333 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:07.349 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:07.350 INFO:teuthology.orchestra.run.smithi112.stdout:933 write 0x96b1a4b thru 0x96c0d0c (0xf2c2 bytes) 2024-09-04T21:54:07.357 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.357 INFO:teuthology.orchestra.run.smithi112.stdout:935 read 0x7015a1f thru 0x7023574 (0xdb56 bytes) 2024-09-04T21:54:07.357 INFO:teuthology.orchestra.run.smithi112.stdout:938 read 0x4db93d8 thru 0x4dc7af7 (0xe720 bytes) 2024-09-04T21:54:07.358 INFO:teuthology.orchestra.run.smithi112.stdout:939 read 0x5beafdd thru 0x5bf71c3 (0xc1e7 bytes) 2024-09-04T21:54:07.358 INFO:teuthology.orchestra.run.smithi112.stdout:940 trunc from 0x9e6d836 to 0x9ca2cc2 2024-09-04T21:54:07.367 INFO:teuthology.orchestra.run.smithi112.stdout:941 write 0x923de77 thru 0x9240f2d (0x30b7 bytes) 2024-09-04T21:54:07.374 INFO:teuthology.orchestra.run.smithi112.stdout:942 trunc from 0x9ca2cc2 to 0x9a9bfbb 2024-09-04T21:54:07.394 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:07.395+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:07.398 INFO:teuthology.orchestra.run.smithi112.stdout:945 write 0xad0b4b0 thru 0xad186f7 (0xd248 bytes) 2024-09-04T21:54:07.406 INFO:teuthology.orchestra.run.smithi112.stdout:948 write 0x283b7f0 thru 0x284b39d (0xfbae bytes) 2024-09-04T21:54:07.421 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:54:07.422 INFO:teuthology.orchestra.run.smithi112.stdout:949 writesame 0x8b98029 thru 0x8b9c7d1 data_size 0x5(0x47a9 bytes) 2024-09-04T21:54:07.448 INFO:teuthology.orchestra.run.smithi112.stdout:950 write 0xe3f099a thru 0xe3f67ec (0x5e53 bytes) 2024-09-04T21:54:07.455 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.455 INFO:teuthology.orchestra.run.smithi112.stdout:952 write 0x6348f63 thru 0x6356cc5 (0xdd63 bytes) 2024-09-04T21:54:07.460 INFO:teuthology.orchestra.run.smithi112.stdout:953 read 0x2e521d2 thru 0x2e58137 (0x5f66 bytes) 2024-09-04T21:54:07.460 INFO:teuthology.orchestra.run.smithi112.stdout:954 writesame 0xbe59889 thru 0xbe5bcc3 data_size 0x1(0x243b bytes) 2024-09-04T21:54:07.485 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:07.484+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:07.489 INFO:teuthology.orchestra.run.smithi112.stdout:955 trunc from 0xe3f67ed to 0x404aced 2024-09-04T21:54:07.513 INFO:teuthology.orchestra.run.smithi112.stdout:956 read 0x16f2428 thru 0x1700a5c (0xe635 bytes) 2024-09-04T21:54:07.514 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.514 INFO:teuthology.orchestra.run.smithi112.stdout:959 read 0x3c32dc3 thru 0x3c37517 (0x4755 bytes) 2024-09-04T21:54:07.514 INFO:teuthology.orchestra.run.smithi112.stdout:961 read 0x1daeeaf thru 0x1db7da3 (0x8ef5 bytes) 2024-09-04T21:54:07.527 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:07.527 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:07.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:991 3.11 3:8e007a0b:::rbd_data.11882e693cd0.0000000000000031:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.824982+0000", 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6996067650000004, 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.008839164, 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.528 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 991 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824982+0000", 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824982+0000", 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824984+0000", 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.824993+0000", 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6109999999999994e-06 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825008+0000", 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5741999999999999e-05 2024-09-04T21:54:07.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825016+0000", 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3119999999999996e-06 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825089+0000", 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2990000000000004e-05 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.825180+0000", 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1741000000000001e-05 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.833719+0000", 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0085383279999999995 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.833761+0000", 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1868e-05 2024-09-04T21:54:07.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.833821+0000", 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0525000000000001e-05 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1023 3.13 3:ca6dc6b8:::rbd_data.11882e693cd0.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.881317+0000", 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6432718529999999, 2024-09-04T21:54:07.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004563554, 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1023 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881317+0000", 2024-09-04T21:54:07.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881317+0000", 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881318+0000", 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.113e-06 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881323+0000", 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9049999999999996e-06 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881335+0000", 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1952e-05 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881341+0000", 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4400000000000002e-06 2024-09-04T21:54:07.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881393+0000", 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1218000000000003e-05 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.881441+0000", 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8801999999999998e-05 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.885784+0000", 2024-09-04T21:54:07.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043429740000000003 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.885815+0000", 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0895999999999997e-05 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.885880+0000", 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5253999999999996e-05 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1028 3.1a 3:5e1a43f5:::rbd_data.11882e693cd0.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906748+0000", 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6178412240000002, 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034791599999999998, 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.539 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1028 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906748+0000", 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906748+0000", 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906750+0000", 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.396e-06 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906760+0000", 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8439999999999993e-06 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906791+0000", 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0735999999999998e-05 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906806+0000", 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5174000000000001e-05 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907594+0000", 2024-09-04T21:54:07.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078809400000000003 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907833+0000", 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023909499999999999 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908124+0000", 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029064500000000002 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909801+0000", 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001677886 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909975+0000", 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000173785 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.910116+0000", 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014066600000000001 2024-09-04T21:54:07.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.910227+0000", 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110839 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1031 3.14 3:28c51851:::rbd_data.11882e693cd0.000000000000000f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906844+0000", 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.617744514, 2024-09-04T21:54:07.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056239519999999998, 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1031 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906844+0000", 2024-09-04T21:54:07.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906844+0000", 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906846+0000", 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.24e-06 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906853+0000", 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2289999999999997e-06 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906871+0000", 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8726999999999999e-05 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906886+0000", 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4116e-05 2024-09-04T21:54:07.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908453+0000", 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015676469999999999 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908657+0000", 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020391800000000001 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912080+0000", 2024-09-04T21:54:07.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003422834 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912235+0000", 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015526599999999999 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912468+0000", 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023297500000000001 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.550 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:1026 3.1b e27/22 3:da206786:::rbd_data.11882e693cd0.0000000000000007:head v 27'7, mlcod=27'5)", 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906982+0000", 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6176066320000002, 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048470600000000003, 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906982+0000", 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906982+0000", 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906984+0000", 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0770000000000002e-06 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906999+0000", 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4528999999999999e-05 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907758+0000", 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075932899999999999 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907801+0000", 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2595999999999997e-05 2024-09-04T21:54:07.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909170+0000", 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013696909999999999 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909226+0000", 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5195999999999998e-05 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911551+0000", 2024-09-04T21:54:07.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023256380000000001 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911829+0000", 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027800399999999999 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1033 3.13 3:cd8fd6b8:::rbd_data.11882e693cd0.0000000000000011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.907646+0000", 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.616943054, 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038967009999999998, 2024-09-04T21:54:07.555 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1033 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.556 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907646+0000", 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907646+0000", 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907648+0000", 2024-09-04T21:54:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2670000000000001e-06 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907658+0000", 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7170000000000003e-06 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907688+0000", 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0620000000000002e-05 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907729+0000", 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0272999999999999e-05 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907889+0000", 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016074199999999999 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908086+0000", 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019644500000000001 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911170+0000", 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030837199999999999 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911320+0000", 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015022500000000001 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911542+0000", 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000222692 2024-09-04T21:54:07.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1034 3.f 3:f5618e06:::rbd_data.11882e693cd0.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.907776+0000", 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6168126479999998, 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037740719999999998, 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1034 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907776+0000", 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907776+0000", 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907778+0000", 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7239999999999999e-06 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907786+0000", 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8429999999999993e-06 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907808+0000", 2024-09-04T21:54:07.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2116e-05 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907824+0000", 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6390000000000001e-05 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908068+0000", 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000244229 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908284+0000", 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021556799999999999 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911210+0000", 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029255470000000001 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911337+0000", 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127043 2024-09-04T21:54:07.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911550+0000", 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021361199999999999 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:1027 3.1c e27/22 3:3c9626f4:::rbd_data.11882e693cd0.0000000000000008:head v 27'8, mlcod=26'6)", 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.907886+0000", 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6167027019999995, 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039703899999999999, 2024-09-04T21:54:07.566 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907886+0000", 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907886+0000", 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907887+0000", 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.314e-06 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907921+0000", 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3155999999999999e-05 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907941+0000", 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0613000000000001e-05 2024-09-04T21:54:07.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907956+0000", 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4693e-05 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908963+0000", 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001007574 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909022+0000", 2024-09-04T21:54:07.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8729e-05 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911319+0000", 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022969219999999999 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911856+0000", 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053738899999999999 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:1035 3.1b e27/22 3:d8cf32f8:::rbd_data.11882e693cd0.0000000000000013:head v 27'8, mlcod=27'5)", 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.908248+0000", 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6163403179999998, 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055671970000000003, 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.571 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908248+0000", 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908248+0000", 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908250+0000", 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.784e-06 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908264+0000", 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.418e-05 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908294+0000", 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9689999999999999e-05 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.908313+0000", 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8859999999999999e-05 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909829+0000", 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015164829999999999 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909897+0000", 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8002000000000004e-05 2024-09-04T21:54:07.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913682+0000", 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037848320000000001 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913816+0000", 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000133366 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1041 3.13 3:ca6dc6b8:::rbd_data.11882e693cd0.0000000000000021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.911934+0000", 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6126548060000001, 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003696436, 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1041 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.576 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911934+0000", 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911934+0000", 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911937+0000", 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.554e-06 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911949+0000", 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2806000000000001e-05 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911975+0000", 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6040000000000001e-05 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.911995+0000", 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9887000000000001e-05 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912603+0000", 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060750599999999995 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912805+0000", 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000202635 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915238+0000", 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024326539999999998 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915377+0000", 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013919900000000001 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915630+0000", 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025315500000000002 2024-09-04T21:54:07.580 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1337 3.3 3:c65d6ddd:::rbd_data.1188e7c968d9.000000000000010c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.853530+0000", 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6710585409999998, 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004973084, 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1337 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853530+0000", 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853530+0000", 2024-09-04T21:54:07.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853533+0000", 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.61e-06 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853545+0000", 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1946e-05 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853594+0000", 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9095999999999999e-05 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853615+0000", 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1146999999999999e-05 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853787+0000", 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017151199999999999 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853983+0000", 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019677900000000001 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854231+0000", 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024740100000000001 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855109+0000", 2024-09-04T21:54:07.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00087801800000000005 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858414+0000", 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033049630000000002 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858469+0000", 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4981000000000001e-05 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858503+0000", 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4631000000000001e-05 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1415 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 4030464~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.55500404, 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003816645, 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.587 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1415 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969585+0000", 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3900000000000005e-07 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969588+0000", 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6079999999999998e-06 2024-09-04T21:54:07.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969595+0000", 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0820000000000001e-06 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.969601+0000", 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5389999999999998e-06 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972842+0000", 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032410770000000002 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.972955+0000", 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011378 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.973401+0000", 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044601999999999999 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1520 3.3 3:c485019f:::rbd_data.1188e7c968d9.00000000000000cc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.182672+0000", 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.3419171690000002, 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0087245150000000004, 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.592 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1520 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182672+0000", 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182672+0000", 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182673+0000", 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4699999999999999e-06 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182679+0000", 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3929999999999996e-06 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182692+0000", 2024-09-04T21:54:07.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2317000000000001e-05 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182698+0000", 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9909999999999999e-06 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182785+0000", 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7075000000000002e-05 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182999+0000", 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000214446 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.184586+0000", 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015862090000000001 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191212+0000", 2024-09-04T21:54:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066266149999999998 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191238+0000", 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6319999999999999e-05 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191317+0000", 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8943000000000006e-05 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.191396+0000", 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8736e-05 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1538 3.1a 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.198855+0000", 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.3257341719999998, 2024-09-04T21:54:07.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054839449999999996, 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1538 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.198855+0000", 2024-09-04T21:54:07.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.198855+0000", 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.198856+0000", 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6020000000000001e-06 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199398+0000", 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054148200000000003 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199418+0000", 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0542000000000001e-05 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199427+0000", 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4980000000000006e-06 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199489+0000", 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1908000000000004e-05 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.199648+0000", 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000159841 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.200091+0000", 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000442557 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.202029+0000", 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00193797 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.204237+0000", 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022081589999999999 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.204293+0000", 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5475e-05 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.204339+0000", 2024-09-04T21:54:07.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5911000000000001e-05 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1613 3.3 3:c7196aef:::rbd_data.1188e7c968d9.00000000000001a5:head [set-alloc-hint object_size 524288 write_size 524288,write 425984~22672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.344480+0000", 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.1801089210000004, 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036000229999999999, 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.605 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1613 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344480+0000", 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344480+0000", 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344481+0000", 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.525e-06 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344514+0000", 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2913000000000001e-05 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344537+0000", 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2838000000000001e-05 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344546+0000", 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4449999999999996e-06 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344638+0000", 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1953999999999994e-05 2024-09-04T21:54:07.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344781+0000", 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014343900000000001 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344864+0000", 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2782999999999997e-05 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.345530+0000", 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066639100000000001 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.348026+0000", 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024954130000000001 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.348056+0000", 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0611999999999998e-05 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.348080+0000", 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3710000000000002e-05 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1749 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.511131+0000", 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0134574729999999, 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036212409999999999, 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.611 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1749 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511131+0000", 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511131+0000", 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511132+0000", 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511134+0000", 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2079999999999999e-06 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511146+0000", 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.224e-05 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511158+0000", 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1039999999999999e-05 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.514525+0000", 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033672870000000001 2024-09-04T21:54:07.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.514610+0000", 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4993000000000004e-05 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.514753+0000", 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014277300000000001 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1750 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 3735552~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.511166+0000", 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0134232110000001, 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039229080000000001, 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1750 2024-09-04T21:54:07.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511166+0000", 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511166+0000", 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511166+0000", 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7899999999999999e-07 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511168+0000", 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3750000000000001e-06 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511175+0000", 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6680000000000004e-06 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511180+0000", 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0950000000000003e-06 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.514819+0000", 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003639279 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.514916+0000", 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6138999999999996e-05 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515088+0000", 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017287300000000001 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1751 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 3899392~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.511187+0000", 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0134017860000002, 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041522699999999996, 2024-09-04T21:54:07.620 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1751 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511187+0000", 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511187+0000", 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511187+0000", 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2599999999999998e-07 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511189+0000", 2024-09-04T21:54:07.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9180000000000001e-06 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511197+0000", 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2470000000000002e-06 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511201+0000", 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8280000000000002e-06 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515117+0000", 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039152270000000003 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515199+0000", 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2421000000000003e-05 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515339+0000", 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014020300000000001 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1752 3.11 3:89c6d28c:::rbd_data.11882e693cd0.0000000000000002:3 [read 4063232~22058] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.511209+0000", 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0133793779999998, 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0046829999999999997, 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:07.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1752 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511209+0000", 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511209+0000", 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511210+0000", 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.08e-07 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511212+0000", 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511225+0000", 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3161e-05 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.511231+0000", 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3859999999999998e-06 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515389+0000", 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041582729999999997 2024-09-04T21:54:07.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515506+0000", 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117074 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.515892+0000", 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038655100000000001 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7309 3.1a 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:06.975844+0000", 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.54874451300000004, 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0051914969999999998, 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7309 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.630 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.975844+0000", 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.975844+0000", 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:07.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.975846+0000", 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.976198+0000", 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035234200000000001 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.976219+0000", 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.056e-05 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.976228+0000", 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2180000000000005e-06 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.976349+0000", 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012078000000000001 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.976513+0000", 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016405400000000001 2024-09-04T21:54:07.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.976960+0000", 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044733000000000001 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.978862+0000", 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001902017 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.980935+0000", 2024-09-04T21:54:07.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020725729999999999 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.980994+0000", 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8909000000000001e-05 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.981036+0000", 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1968000000000002e-05 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.635 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.636 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:07.636 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:07.636 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:07.638 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:07.638 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:07.635+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:07.639 INFO:teuthology.orchestra.run.smithi112.stdout:962 writesame 0x54527c7 thru 0x5459420 data_size 0x45(0x6c5a bytes) 2024-09-04T21:54:07.639 INFO:teuthology.orchestra.run.smithi112.stdout:964 read 0x21ea4fe thru 0x21f7f23 (0xda26 bytes) 2024-09-04T21:54:07.639 INFO:teuthology.orchestra.run.smithi112.stdout:965 punch from 0x2c95777 to 0x2ca5299, (0xfb22 bytes) 2024-09-04T21:54:07.639 INFO:teuthology.orchestra.run.smithi112.stdout:966 read 0x4f1df5b thru 0x4f28ada (0xab80 bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:969 writesame 0xe2de523 thru 0xe2eadec data_size 0x27(0xc8ca bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:970 trunc from 0xe2eaded to 0xda6530b 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:971 write 0x1bc8142 thru 0x1bd53f3 (0xd2b2 bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:972 read 0x2502401 thru 0x25089ea (0x65ea bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:973 punch from 0xe6e41a to 0xe7ca19, (0xe5ff bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:974 trunc from 0xda6530b to 0x8f3e667 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:976 read 0x80255e7 thru 0x8030c33 (0xb64d bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:979 trunc from 0x8f3e667 to 0xba4e151 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:981 punch from 0x80ed014 to 0x80f0877, (0x3863 bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:983 read 0xb62bc68 thru 0xb62e4be (0x2857 bytes) 2024-09-04T21:54:07.640 INFO:teuthology.orchestra.run.smithi112.stdout:984 trunc from 0xba4e151 to 0x5a8d40b 2024-09-04T21:54:07.653 INFO:teuthology.orchestra.run.smithi112.stdout:986 read 0x465ebb0 thru 0x465f91f (0xd70 bytes) 2024-09-04T21:54:07.678 INFO:teuthology.orchestra.run.smithi112.stdout:987 write 0x8e92a62 thru 0x8e9d1b6 (0xa755 bytes) 2024-09-04T21:54:07.686 INFO:teuthology.orchestra.run.smithi112.stdout:988 punch from 0x6441f5e to 0x644cd78, (0xae1a bytes) 2024-09-04T21:54:07.686 INFO:teuthology.orchestra.run.smithi112.stdout:989 punch from 0x5e0f2ce to 0x5e1907e, (0x9db0 bytes) 2024-09-04T21:54:07.709 INFO:teuthology.orchestra.run.smithi112.stdout:990 trunc from 0x8e9d1b7 to 0x73c301 2024-09-04T21:54:07.738 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:07.739+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:07.756 INFO:teuthology.orchestra.run.smithi112.stdout:992 read 0x3f63ce thru 0x40028a (0x9ebd bytes) 2024-09-04T21:54:07.757 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.757 INFO:teuthology.orchestra.run.smithi112.stdout:994 punch from 0x18f91e to 0x191b7b, (0x225d bytes) 2024-09-04T21:54:07.801 INFO:teuthology.orchestra.run.smithi112.stdout:996 write 0xbfa076c thru 0xbfafba0 (0xf435 bytes) 2024-09-04T21:54:07.809 INFO:teuthology.orchestra.run.smithi112.stdout:997 trunc from 0xbfafba1 to 0xbc5141b 2024-09-04T21:54:07.819 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:07.820 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:07.820 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:07.820 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:07.824 INFO:teuthology.orchestra.run.smithi112.stdout:999 punch from 0x1bdd69f to 0x1be3756, (0x60b7 bytes) 2024-09-04T21:54:07.824 INFO:teuthology.orchestra.run.smithi112.stdout:1001 writesame 0x5686b07 thru 0x56901df data_size 0x1(0x96d9 bytes) 2024-09-04T21:54:07.835 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:07.863 INFO:teuthology.orchestra.run.smithi112.stdout:1002 write 0xb74fc8b thru 0xb7575a7 (0x791d bytes) 2024-09-04T21:54:07.878 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:07.879+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:07.889 INFO:teuthology.orchestra.run.smithi112.stdout:1003 writesame 0xe019449 thru 0xe01bf03 data_size 0x1(0x2abb bytes) 2024-09-04T21:54:07.889 INFO:teuthology.orchestra.run.smithi112.stdout:1005 write 0x7cde353 thru 0x7ce99c8 (0xb676 bytes) 2024-09-04T21:54:07.899 INFO:teuthology.orchestra.run.smithi112.stdout:1006 read 0x57e0c88 thru 0x57e14fc (0x875 bytes) 2024-09-04T21:54:07.899 INFO:teuthology.orchestra.run.smithi112.stdout:1007 writesame 0xceaeda9 thru 0xceaff9c data_size 0x6(0x11f4 bytes) 2024-09-04T21:54:07.911 INFO:teuthology.orchestra.run.smithi112.stdout:1008 trunc from 0xe01bf04 to 0xbc4561e 2024-09-04T21:54:07.930 INFO:teuthology.orchestra.run.smithi112.stdout:1009 read 0x7c9cf15 thru 0x7ca3d9f (0x6e8b bytes) 2024-09-04T21:54:07.930 INFO:teuthology.orchestra.run.smithi112.stdout:1010 writesame 0x7784dc8 thru 0x7787cbf data_size 0x6(0x2ef8 bytes) 2024-09-04T21:54:07.941 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:07.943+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:07.947 INFO:teuthology.orchestra.run.smithi112.stdout:1011 write 0xcc0fde1 thru 0xcc16dcc (0x6fec bytes) 2024-09-04T21:54:07.953 INFO:teuthology.orchestra.run.smithi112.stdout:truncating to largest ever: 0xedf6026 2024-09-04T21:54:07.959 INFO:teuthology.orchestra.run.smithi112.stdout:1012 trunc from 0xcc16dcd to 0xedf6026 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1015 read 0x14c1222 thru 0x14ca723 (0x9502 bytes) 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1018 punch from 0x436f617 to 0x4373653, (0x403c bytes) 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1019 read 0xafb96fe thru 0xafbf67b (0x5f7e bytes) 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1022 punch from 0xcd5aa4b to 0xcd603df, (0x5994 bytes) 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1023 punch from 0x17f21dd to 0x17f7a3c, (0x585f bytes) 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1024 punch from 0x793261f to 0x7936a2b, (0x440c bytes) 2024-09-04T21:54:07.972 INFO:teuthology.orchestra.run.smithi112.stdout:1025 writesame 0x25fc0b0 thru 0x260bfa9 data_size 0x8a(0xfefa bytes) 2024-09-04T21:54:07.980 INFO:teuthology.orchestra.run.smithi112.stdout:1026 punch from 0x10cce4b to 0x10d78ae, (0xaa63 bytes) 2024-09-04T21:54:07.980 INFO:teuthology.orchestra.run.smithi112.stdout:1027 clone 4 order 21 su 8192 sc 9 2024-09-04T21:54:08.006 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:08.006 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:08.006 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:08.006 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:08.006 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:08.020 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:08.043 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:08.044+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:08.144 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:08.143+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:08.181 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 21.375194390000001, 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068597409999999999, 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.182 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4208", 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/1349276815", 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804261+0000", 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0230000000000001e-06 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804276+0000", 2024-09-04T21:54:08.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4304000000000001e-05 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804309+0000", 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3463000000000001e-05 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804328+0000", 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9307e-05 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804489+0000", 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000160672 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804499+0000", 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0283e-05 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.807660+0000", 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003160969 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.807887+0000", 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022673800000000001 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.808103+0000", 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021598900000000001 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.809574+0000", 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014714940000000001 2024-09-04T21:54:08.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.810928+0000", 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001353344 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.811009+0000", 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1406e-05 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.811118+0000", 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108749 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.370837085000002, 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0075919359999999996, 2024-09-04T21:54:08.189 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4208", 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/1349276815", 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808619+0000", 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808652+0000", 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2842999999999997e-05 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808686+0000", 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4205999999999997e-05 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808703+0000", 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7597000000000001e-05 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808872+0000", 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016823599999999999 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808889+0000", 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7716000000000001e-05 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813206+0000", 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043164680000000004 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813380+0000", 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017376199999999999 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813595+0000", 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021576899999999999 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.815122+0000", 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015264339999999999 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816053+0000", 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00093072599999999999 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816141+0000", 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8462999999999996e-05 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816207+0000", 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6429999999999999e-05 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1027 3.1c 3:3c9626f4:::rbd_data.11882e693cd0.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:08.197 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.2727311740000005, 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056794660000000002, 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.198 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1027 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906724+0000", 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4739999999999999e-06 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906736+0000", 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2e-05 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906768+0000", 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2106000000000001e-05 2024-09-04T21:54:08.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906789+0000", 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1313000000000001e-05 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906990+0000", 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020057300000000001 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907213+0000", 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022263099999999999 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907521+0000", 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000308485 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909663+0000", 2024-09-04T21:54:08.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021414559999999999 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912156+0000", 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002493033 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912289+0000", 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013296900000000001 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912401+0000", 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112426 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1187 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578279+0000", 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6011735209999998, 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052155020000000003, 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.205 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1187 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578279+0000", 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578279+0000", 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578281+0000", 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4339999999999999e-06 2024-09-04T21:54:08.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578286+0000", 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8540000000000002e-06 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578301+0000", 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4379000000000001e-05 2024-09-04T21:54:08.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578308+0000", 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3590000000000002e-06 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:08.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9577999999999998e-05 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578459+0000", 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0947999999999997e-05 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578813+0000", 2024-09-04T21:54:08.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035448199999999999 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583085+0000", 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042723209999999999 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583225+0000", 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013905800000000001 2024-09-04T21:54:08.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583374+0000", 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014990700000000001 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583495+0000", 2024-09-04T21:54:08.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120182 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1188 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6011128939999999, 2024-09-04T21:54:08.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054046620000000002, 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1188 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1900000000000002e-07 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578348+0000", 2024-09-04T21:54:08.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3560000000000004e-06 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578355+0000", 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.114e-06 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578359+0000", 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4660000000000001e-06 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.579323+0000", 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096376800000000005 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.579621+0000", 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029837100000000002 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.580410+0000", 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078867200000000003 2024-09-04T21:54:08.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583596+0000", 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031858160000000002 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583615+0000", 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9224000000000001e-05 2024-09-04T21:54:08.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583678+0000", 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2613999999999994e-05 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583744+0000", 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6642000000000003e-05 2024-09-04T21:54:08.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1189 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6010847950000002, 2024-09-04T21:54:08.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056526730000000004, 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1189 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6100000000000002e-07 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578370+0000", 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2479999999999999e-06 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578375+0000", 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3819999999999997e-06 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578380+0000", 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0509999999999996e-06 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.581409+0000", 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030287550000000002 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.581628+0000", 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021934499999999999 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.582249+0000", 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062055900000000002 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583761+0000", 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015120960000000001 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583845+0000", 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4357000000000006e-05 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583945+0000", 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9550999999999996e-05 2024-09-04T21:54:08.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584021+0000", 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5968000000000002e-05 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.230 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1248 3.16 3:6ce50ae3:::rbd_data.1188e7c968d9.0000000000000087:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.5019614250000002, 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013362881, 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.231 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1248 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:08.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677492+0000", 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.035e-06 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677497+0000", 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5050000000000001e-06 2024-09-04T21:54:08.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677507+0000", 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8679999999999994e-06 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677514+0000", 2024-09-04T21:54:08.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7049999999999999e-06 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677595+0000", 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0308000000000004e-05 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677748+0000", 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000153425 2024-09-04T21:54:08.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690686+0000", 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.01293805 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690748+0000", 2024-09-04T21:54:08.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2189999999999999e-05 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690854+0000", 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105795 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1303 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.373499647, 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0064365610000000004, 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.238 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1303 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:08.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805954+0000", 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.285e-06 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805960+0000", 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0460000000000002e-06 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805975+0000", 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4304999999999999e-05 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805981+0000", 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5919999999999997e-06 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806084+0000", 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103027 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806256+0000", 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017127900000000001 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806853+0000", 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00059703999999999996 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.811952+0000", 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050998559999999998 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812130+0000", 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017728899999999999 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812269+0000", 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013945900000000001 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812390+0000", 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120383 2024-09-04T21:54:08.246 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1304 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3734425769999996, 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066469160000000001, 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.247 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1304 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:08.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806011+0000", 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4000000000000001e-07 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806014+0000", 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.444e-06 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806030+0000", 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5480000000000001e-05 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806035+0000", 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0649999999999998e-06 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.807688+0000", 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016533559999999999 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.807956+0000", 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000268182 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.809072+0000", 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001115659 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812090+0000", 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003018097 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812415+0000", 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032458399999999999 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812546+0000", 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013182600000000001 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812657+0000", 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011048300000000001 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1305 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3733903290000002, 2024-09-04T21:54:08.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073242790000000004, 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1305 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806063+0000", 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4400000000000001e-07 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806065+0000", 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5450000000000002e-06 2024-09-04T21:54:08.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806072+0000", 2024-09-04T21:54:08.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-04T21:54:08.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806075+0000", 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4960000000000001e-06 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.810223+0000", 2024-09-04T21:54:08.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041482749999999999 2024-09-04T21:54:08.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.810500+0000", 2024-09-04T21:54:08.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027680199999999998 2024-09-04T21:54:08.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.811129+0000", 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062907200000000005 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812801+0000", 2024-09-04T21:54:08.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001671753 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813238+0000", 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043750000000000001 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813321+0000", 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2145999999999996e-05 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813387+0000", 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6045999999999998e-05 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1334 3.c 3:303b0518:::rbd_data.1188e7c968d9.0000000000000101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3268586659999997, 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056592250000000004, 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.264 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1334 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852595+0000", 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.322e-06 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852602+0000", 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3119999999999999e-06 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852623+0000", 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1106000000000001e-05 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852636+0000", 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3692e-05 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852743+0000", 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106747 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852884+0000", 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000140755 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853074+0000", 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019004499999999999 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854595+0000", 2024-09-04T21:54:08.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001521175 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858170+0000", 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035752470000000001 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858216+0000", 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5948000000000001e-05 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858253+0000", 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6875999999999997e-05 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1445 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.032341+0000", 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.1471118149999997, 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050597009999999998, 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.270 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1445 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032341+0000", 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032341+0000", 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032342+0000", 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.004e-06 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032347+0000", 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7759999999999997e-06 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032358+0000", 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1243000000000001e-05 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.032365+0000", 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5499999999999997e-06 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.034860+0000", 2024-09-04T21:54:08.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024945969999999999 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.034958+0000", 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8361000000000002e-05 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.035285+0000", 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032705899999999999 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.036215+0000", 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092982399999999999 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.037323+0000", 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001107532 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.037358+0000", 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4959999999999997e-05 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.037401+0000", 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2794999999999999e-05 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1518 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.002088079, 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0049234650000000001, 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.276 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1518 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177365+0000", 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1699999999999999e-07 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177369+0000", 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.092e-06 2024-09-04T21:54:08.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177375+0000", 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.959e-06 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.177378+0000", 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3179999999999998e-06 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.179942+0000", 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025639339999999999 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.180036+0000", 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4159000000000004e-05 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.180366+0000", 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032972299999999997 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.181602+0000", 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012356209999999999 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182229+0000", 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062770600000000001 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182263+0000", 2024-09-04T21:54:08.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3506000000000001e-05 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.182288+0000", 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5029999999999999e-05 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1785 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.632683643, 2024-09-04T21:54:08.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048312010000000002, 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1785 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546769+0000", 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8299999999999998e-07 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546773+0000", 2024-09-04T21:54:08.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4750000000000002e-06 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546777+0000", 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4170000000000002e-06 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.546780+0000", 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1650000000000002e-06 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.549526+0000", 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027455219999999998 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.549627+0000", 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100929 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.550013+0000", 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038640099999999998 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551014+0000", 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010010990000000001 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551542+0000", 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00052794799999999998 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551576+0000", 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3466000000000001e-05 2024-09-04T21:54:08.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.551600+0000", 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4396000000000002e-05 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7382 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.175714+0000", 2024-09-04T21:54:08.288 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0037389750000001, 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0049105329999999999, 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7382 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.175714+0000", 2024-09-04T21:54:08.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.175714+0000", 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.175715+0000", 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8299999999999995e-07 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.175719+0000", 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8380000000000001e-06 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.175728+0000", 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5080000000000005e-06 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.175736+0000", 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9130000000000003e-06 2024-09-04T21:54:08.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.177423+0000", 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016869579999999999 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.177563+0000", 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014056800000000001 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.178228+0000", 2024-09-04T21:54:08.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066449 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.179555+0000", 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001327134 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.180453+0000", 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00089823500000000005 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.180552+0000", 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8814000000000006e-05 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.180624+0000", 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2092e-05 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7494 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:08.294 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.411570+0000", 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.76788309899999996, 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005063349, 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7494 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411570+0000", 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411570+0000", 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411571+0000", 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.513e-06 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411579+0000", 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6590000000000007e-06 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411599+0000", 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0780000000000001e-05 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411611+0000", 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1235e-05 2024-09-04T21:54:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411687+0000", 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6163999999999998e-05 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.411776+0000", 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9352999999999996e-05 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.412085+0000", 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030919999999999998 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.413759+0000", 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001673332 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.416485+0000", 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027265290000000001 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.416571+0000", 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5891999999999997e-05 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.416633+0000", 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1692000000000005e-05 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7670 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.733317+0000", 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.44613560800000002, 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048500039999999998, 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7670 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.301 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733317+0000", 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733317+0000", 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733319+0000", 2024-09-04T21:54:08.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.646e-06 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733326+0000", 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9070000000000001e-06 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733340+0000", 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4056999999999999e-05 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733347+0000", 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5610000000000001e-06 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733407+0000", 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9574000000000003e-05 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733474+0000", 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6742000000000006e-05 2024-09-04T21:54:08.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.733723+0000", 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024936899999999999 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735176+0000", 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014527159999999999 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738029+0000", 2024-09-04T21:54:08.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028529110000000001 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738105+0000", 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6482000000000004e-05 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738167+0000", 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2038999999999993e-05 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.306 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7673 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735557+0000", 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.44389536299999999, 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005252278, 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7673 2024-09-04T21:54:08.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735557+0000", 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735557+0000", 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735558+0000", 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3300000000000002e-07 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735561+0000", 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.23e-06 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735575+0000", 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4422e-05 2024-09-04T21:54:08.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735581+0000", 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736854+0000", 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012722320000000001 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736966+0000", 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112237 2024-09-04T21:54:08.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.737490+0000", 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00052420400000000001 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739475+0000", 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019846130000000001 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740683+0000", 2024-09-04T21:54:08.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012080019999999999 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740764+0000", 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1564999999999996e-05 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740810+0000", 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5373000000000001e-05 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7674 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735590+0000", 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.44386225499999998, 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0058458, 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7674 2024-09-04T21:54:08.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735590+0000", 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735590+0000", 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735591+0000", 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8700000000000001e-07 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735595+0000", 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.912e-06 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735602+0000", 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1149999999999996e-06 2024-09-04T21:54:08.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735606+0000", 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0119999999999999e-06 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738398+0000", 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002792414 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738503+0000", 2024-09-04T21:54:08.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105046 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738961+0000", 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045731800000000001 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739909+0000", 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094838400000000003 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741310+0000", 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014005860000000001 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741379+0000", 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9648999999999997e-05 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741436+0000", 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6977e-05 2024-09-04T21:54:08.318 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7678 3.c 3:30b5c6f9:::rbd_data.1188b1294ada.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.740524+0000", 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.43892863700000001, 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006006323, 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.319 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7678 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740524+0000", 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740524+0000", 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740526+0000", 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1440000000000001e-06 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741421+0000", 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000894892 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741435+0000", 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3991e-05 2024-09-04T21:54:08.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741442+0000", 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2960000000000001e-06 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741519+0000", 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6983999999999993e-05 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741656+0000", 2024-09-04T21:54:08.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000136468 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741963+0000", 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030737899999999999 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.744947+0000", 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029836860000000002 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.746457+0000", 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00151012 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.746501+0000", 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3680999999999997e-05 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.746530+0000", 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9682000000000001e-05 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.325 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.325 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:08.326 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:08.327 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:08.328+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:08.427 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:08.427+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7824 3.a 3:572c7a4a:::rbd_header.1188e197bdd2:head [create,call rbd.create in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.00036290399999999999, 2024-09-04T21:54:08.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042473700000000001, 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7824 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509758+0000", 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4800000000000001e-06 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509771+0000", 2024-09-04T21:54:08.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3073e-05 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509806+0000", 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4913999999999998e-05 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509828+0000", 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2209e-05 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509960+0000", 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131874 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510196+0000", 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023616500000000001 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 1 2024-09-04T21:54:08.513 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:08.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:08.528 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:08.528+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:08.556 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone3 from 0xedf6026 (overlap 0x73c301) to 0x172700 2024-09-04T21:54:08.629 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:08.628+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:08.679 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-09-04T21:54:08.696 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:08.696 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:08.696 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:08.696 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:08.696 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:08.710 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:08.754 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:08.755+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:08.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:08.830 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:08.831+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:08.837 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:08.837 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:08.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1045 3.10 3:08ed8dc3:::rbd_data.11882e693cd0.000000000000002f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.9212707440000001, 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054299810000000004, 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1045 2024-09-04T21:54:08.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912477+0000", 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8369999999999999e-06 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912492+0000", 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.428e-05 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912529+0000", 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7673000000000003e-05 2024-09-04T21:54:08.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912547+0000", 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7689e-05 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912710+0000", 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162526 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912993+0000", 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028371199999999998 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913324+0000", 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033046099999999999 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915482+0000", 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021584989999999999 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917665+0000", 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021827449999999998 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917778+0000", 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000113063 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917904+0000", 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012549600000000001 2024-09-04T21:54:08.843 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1101 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.4134147019999999, 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0067943780000000002, 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.844 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1101 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:08.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420333+0000", 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6169999999999998e-06 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420347+0000", 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-05 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420381+0000", 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4122999999999998e-05 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420398+0000", 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6286999999999998e-05 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420568+0000", 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017088199999999999 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420589+0000", 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0513000000000002e-05 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424142+0000", 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035527359999999999 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424352+0000", 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021066300000000001 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424715+0000", 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000362233 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426277+0000", 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015626069999999999 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426969+0000", 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069220800000000004 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427060+0000", 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1057000000000006e-05 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427124+0000", 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3522000000000006e-05 2024-09-04T21:54:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1102 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_stripe_unit_count in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.4063856589999997, 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026178130000000001, 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1102 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427361+0000", 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.384e-06 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427371+0000", 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0086e-05 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427394+0000", 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2643e-05 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427406+0000", 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2247e-05 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427515+0000", 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010909899999999999 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427646+0000", 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013051299999999999 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.428016+0000", 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037050700000000001 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429054+0000", 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010376230000000001 2024-09-04T21:54:08.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429807+0000", 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075308500000000002 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429894+0000", 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7216e-05 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429976+0000", 2024-09-04T21:54:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2410000000000005e-05 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1116 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [watch watch cookie 139952113516272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3863073359999998, 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029237209999999998, 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1116 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447439+0000", 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447449+0000", 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8239999999999995e-06 2024-09-04T21:54:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447481+0000", 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1733000000000003e-05 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447497+0000", 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5653000000000001e-05 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447642+0000", 2024-09-04T21:54:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014499599999999999 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447778+0000", 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013675199999999999 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.448006+0000", 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022790500000000001 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.449250+0000", 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001244172 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450163+0000", 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00091221100000000001 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450248+0000", 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5753000000000004e-05 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450361+0000", 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011235600000000001 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1192 3.a 3:52245ef8:::rbd_data.1188e7c968d9.00000000000001c6:head [set-alloc-hint object_size 524288 write_size 524288,write 131042~30] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.2495629949999998, 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027730150000000002, 2024-09-04T21:54:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1192 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584184+0000", 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8629999999999999e-06 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584198+0000", 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3961999999999999e-05 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584231+0000", 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2660000000000002e-05 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584250+0000", 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.908e-05 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584339+0000", 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9456000000000005e-05 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584499+0000", 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000159431 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584675+0000", 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017591899999999999 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586136+0000", 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014611050000000001 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586833+0000", 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069698700000000004 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586903+0000", 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0216000000000007e-05 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586954+0000", 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1335999999999998e-05 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1267 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.1044783899999997, 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026536139999999999, 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1267 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729268+0000", 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3800000000000001e-06 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729279+0000", 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0749e-05 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729309+0000", 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0029999999999999e-05 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729323+0000", 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.411e-05 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729437+0000", 2024-09-04T21:54:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114115 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729610+0000", 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000172821 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729965+0000", 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035526999999999998 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.730787+0000", 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082178999999999998 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731730+0000", 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094278199999999997 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731845+0000", 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114935 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731920+0000", 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4632e-05 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1279 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.0700156109999996, 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027977710000000001, 2024-09-04T21:54:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1279 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763731+0000", 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.537e-06 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763743+0000", 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1426999999999999e-05 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763775+0000", 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1925999999999999e-05 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763789+0000", 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4756999999999999e-05 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763900+0000", 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011079800000000001 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.764069+0000", 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169307 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.764651+0000", 2024-09-04T21:54:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058184199999999995 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766213+0000", 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015619130000000001 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766364+0000", 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000150502 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766453+0000", 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8902999999999999e-05 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766526+0000", 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3858999999999999e-05 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1338 3.10 3:0a516185:::rbd_data.1188e7c968d9.000000000000010d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.9801097990000001, 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050744040000000002, 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1338 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853638+0000", 2024-09-04T21:54:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853649+0000", 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1161999999999999e-05 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853682+0000", 2024-09-04T21:54:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3127000000000003e-05 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853697+0000", 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5318000000000001e-05 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853801+0000", 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010342500000000001 2024-09-04T21:54:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853935+0000", 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000133771 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854097+0000", 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016247399999999999 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855226+0000", 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011287210000000001 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858515+0000", 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032894249999999999 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858634+0000", 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119226 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858709+0000", 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4468999999999995e-05 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1401 3.10 3:0966b599:::rbd_data.1188e7c968d9.0000000000000170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.8709995629999998, 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026384160000000002, 2024-09-04T21:54:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1401 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962748+0000", 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962757+0000", 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1600000000000004e-06 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962781+0000", 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4426e-05 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962793+0000", 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1955e-05 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962928+0000", 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135282 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963165+0000", 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023646100000000001 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963584+0000", 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041887000000000001 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965107+0000", 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015236970000000001 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965129+0000", 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.139e-05 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965258+0000", 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012891199999999999 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965383+0000", 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012537900000000001 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1612 3.10 3:0aa9c1cb:::rbd_data.1188e7c968d9.00000000000001a4:head [set-alloc-hint object_size 524288 write_size 524288,write 426878~31874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.4898569000000004, 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003599587, 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1612 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343891+0000", 2024-09-04T21:54:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.23e-06 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343991+0000", 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010068000000000001 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344028+0000", 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6579000000000003e-05 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344049+0000", 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1002999999999999e-05 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344136+0000", 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7100000000000003e-05 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344241+0000", 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010485200000000001 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344342+0000", 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100721 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.345192+0000", 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085025300000000003 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347429+0000", 2024-09-04T21:54:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022367620000000002 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347462+0000", 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2960999999999998e-05 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347487+0000", 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5446e-05 2024-09-04T21:54:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1615 3.10 3:0aa9c1cb:::rbd_data.1188e7c968d9.00000000000001a4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.4770441920000001, 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029358750000000001, 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1615 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356702+0000", 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1830000000000001e-06 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356711+0000", 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5469999999999997e-06 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356769+0000", 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7896000000000003e-05 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356789+0000", 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0239999999999999e-05 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356866+0000", 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6822000000000004e-05 2024-09-04T21:54:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356941+0000", 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5223999999999997e-05 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.357037+0000", 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6063000000000005e-05 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.357780+0000", 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074262700000000004 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359531+0000", 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001751072 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359588+0000", 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6884e-05 2024-09-04T21:54:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359636+0000", 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8316999999999998e-05 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7222 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [watch watch cookie 139952180803216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.0165754310000001, 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031156790000000001, 2024-09-04T21:54:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7222 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817171+0000", 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0600000000000002e-06 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817180+0000", 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3300000000000005e-06 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817213+0000", 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2750999999999999e-05 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817229+0000", 2024-09-04T21:54:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5498e-05 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817353+0000", 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012423600000000001 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817584+0000", 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000230938 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817997+0000", 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041338900000000001 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.819260+0000", 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012629690000000001 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820086+0000", 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082551100000000002 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820175+0000", 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104999999999995e-05 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820285+0000", 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109892 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7465 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1039991~8585] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.4608130189999999, 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030096060000000002, 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7465 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372934+0000", 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:54:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372947+0000", 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3232e-05 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372962+0000", 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5656000000000001e-05 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372970+0000", 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7500000000000003e-06 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373031+0000", 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1197000000000004e-05 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373143+0000", 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011143599999999999 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373258+0000", 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114842 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.374309+0000", 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010515570000000001 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375856+0000", 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001547185 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375909+0000", 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2335e-05 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375941+0000", 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2110000000000003e-05 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7519 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.3387153430000001, 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054280980000000001, 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7519 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495033+0000", 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7299999999999999e-06 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495047+0000", 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4066000000000001e-05 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495089+0000", 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2700000000000001e-05 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495112+0000", 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2965000000000001e-05 2024-09-04T21:54:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495230+0000", 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117581 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495238+0000", 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.388e-06 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498788+0000", 2024-09-04T21:54:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035494939999999998 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498872+0000", 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4190999999999994e-05 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498983+0000", 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110739 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.499844+0000", 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086077500000000004 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500358+0000", 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051428599999999997 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500414+0000", 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5689999999999997e-05 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500457+0000", 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3492999999999998e-05 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7675 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1179648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.097923929, 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057294420000000004, 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7675 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735829+0000", 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8079999999999994e-06 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736259+0000", 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042982200000000001 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736276+0000", 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7124000000000001e-05 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736287+0000", 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0899e-05 2024-09-04T21:54:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736380+0000", 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3304e-05 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736530+0000", 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014963300000000001 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736871+0000", 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034072099999999999 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739225+0000", 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023541030000000002 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741419+0000", 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021947189999999999 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741495+0000", 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5087999999999997e-05 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741550+0000", 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5220999999999999e-05 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7676 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1164033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0949519759999999, 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00344502, 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7676 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738795+0000", 2024-09-04T21:54:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3180000000000001e-06 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739162+0000", 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036728900000000003 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739179+0000", 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7079999999999999e-05 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739189+0000", 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9930000000000007e-06 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739281+0000", 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1866000000000004e-05 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739489+0000", 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020804800000000001 2024-09-04T21:54:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739790+0000", 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030148600000000001 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741943+0000", 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021526610000000002 2024-09-04T21:54:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742126+0000", 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018295599999999999 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742191+0000", 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5234000000000006e-05 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742237+0000", 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6088999999999999e-05 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7824 3.a 3:572c7a4a:::rbd_header.1188e197bdd2:head [create,call rbd.create in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.32398956499999998, 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030039490000000001, 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7824 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509758+0000", 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4800000000000001e-06 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509771+0000", 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3073e-05 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509806+0000", 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4913999999999998e-05 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509828+0000", 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2209e-05 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509960+0000", 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131874 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510196+0000", 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023616500000000001 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510538+0000", 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034151599999999999 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.511889+0000", 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001351217 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512597+0000", 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070789799999999999 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512680+0000", 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3138000000000003e-05 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512759+0000", 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8465000000000002e-05 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7874 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.25479561899999997, 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0051755769999999998, 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7874 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578951+0000", 2024-09-04T21:54:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9360000000000002e-06 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578958+0000", 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578977+0000", 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9011000000000001e-05 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578987+0000", 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3950000000000007e-06 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579158+0000", 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000171266 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579330+0000", 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017195599999999999 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579662+0000", 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033173499999999999 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.583052+0000", 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033902189999999999 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584018+0000", 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096648399999999998 2024-09-04T21:54:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584074+0000", 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5513999999999997e-05 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584124+0000", 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0441e-05 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7875 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.254603042, 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052294439999999998, 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7875 2024-09-04T21:54:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579142+0000", 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0500000000000002e-07 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579147+0000", 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1120000000000004e-06 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579154+0000", 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.097e-06 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579159+0000", 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4870000000000004e-06 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581460+0000", 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023010230000000001 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581541+0000", 2024-09-04T21:54:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0939999999999994e-05 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581739+0000", 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019828000000000001 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584176+0000", 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002437051 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584259+0000", 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2831999999999996e-05 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584320+0000", 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1204000000000006e-05 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584371+0000", 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0612999999999998e-05 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4488.0:7874 3.a e32/22 ondisk, result = 0)", 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.252463035, 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028556430000000002, 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581283+0000", 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5650000000000001e-06 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581289+0000", 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1940000000000003e-06 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581304+0000", 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5386000000000001e-05 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581314+0000", 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2609999999999999e-06 2024-09-04T21:54:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584004+0000", 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002690539 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584013+0000", 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3459999999999994e-06 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584137+0000", 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012435199999999999 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:08.963 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:08.964 INFO:teuthology.orchestra.run.smithi112.stdout:1028 punch from 0x68623b1 to 0x686ec63, (0xc8b2 bytes) 2024-09-04T21:54:08.964 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:08.965 INFO:teuthology.orchestra.run.smithi112.stdout:1030 punch from 0xc33951c to 0xc33f097, (0x5b7b bytes) 2024-09-04T21:54:08.965 INFO:teuthology.orchestra.run.smithi112.stdout:1031 clone 5 order 19 su 131072 sc 6 2024-09-04T21:54:08.965 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:08.965 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:08.966 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:08.963+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:08.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:08.967 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:08.967 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:08.967 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:08.968 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:08.985 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:08.985 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:09.021 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:09.022 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:09.065 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:09.063+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:09.095 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:09.095 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:09.109 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:09.166 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:09.167+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:09.244 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:09.245 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:09.267 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:09.267+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:09.270 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:09.270 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:09.271 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:09.273 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:09.273 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:09.276 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:09.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:09.375 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:09.376+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:09.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:54:09.453 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:09.453 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:09.469 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:09.470 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:09.467+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:09.570 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:09.571+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:09.574 INFO:teuthology.orchestra.run.smithi112.stdout:leaving image image_client.0-clone4 intact 2024-09-04T21:54:09.643 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:09.643 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:09.660 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:09.708 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:09.708+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:09.771 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:09.772+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:09.778 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:09.778 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:09.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:09.807 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-09-04T21:54:09.815 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:09.815 INFO:teuthology.orchestra.run.smithi112.stdout:1035 write 0x3ea5427 thru 0x3ea93ee (0x3fc8 bytes) 2024-09-04T21:54:09.826 INFO:teuthology.orchestra.run.smithi112.stdout:1037 writesame 0x49d8e87 thru 0x49e7c85 data_size 0x3(0xedff bytes) 2024-09-04T21:54:09.844 INFO:teuthology.orchestra.run.smithi112.stdout:1038 writesame 0x9a35fa8 thru 0x9a3d511 data_size 0x26(0x756a bytes) 2024-09-04T21:54:09.849 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:09.849 INFO:teuthology.orchestra.run.smithi112.stdout:1041 writesame 0x57862c3 thru 0x578a86a data_size 0xc(0x45a8 bytes) 2024-09-04T21:54:09.857 INFO:teuthology.orchestra.run.smithi112.stdout:1045 read 0x1a016ec thru 0x1a05ce5 (0x45fa bytes) 2024-09-04T21:54:09.857 INFO:teuthology.orchestra.run.smithi112.stdout:1046 read 0x849973a thru 0x849c940 (0x3207 bytes) 2024-09-04T21:54:09.857 INFO:teuthology.orchestra.run.smithi112.stdout:1047 write 0x2bc3ad8 thru 0x2bc9736 (0x5c5f bytes) 2024-09-04T21:54:09.863 INFO:teuthology.orchestra.run.smithi112.stdout:1050 read 0x5fc1e51 thru 0x5fcf8d8 (0xda88 bytes) 2024-09-04T21:54:09.863 INFO:teuthology.orchestra.run.smithi112.stdout:1052 write 0x22a3a42 thru 0x22ac836 (0x8df5 bytes) 2024-09-04T21:54:09.871 INFO:teuthology.orchestra.run.smithi112.stdout:1055 read 0x805f191 thru 0x8069421 (0xa291 bytes) 2024-09-04T21:54:09.872 INFO:teuthology.orchestra.run.smithi112.stdout:1056 write 0xe06e5b thru 0xe0f922 (0x8ac8 bytes) 2024-09-04T21:54:09.872 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:09.871+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:09.884 INFO:teuthology.orchestra.run.smithi112.stdout:1057 write 0xdb6e578 thru 0xdb71b87 (0x3610 bytes) 2024-09-04T21:54:09.889 INFO:teuthology.orchestra.run.smithi112.stdout:1058 punch from 0xb293602 to 0xb293d68, (0x766 bytes) 2024-09-04T21:54:09.898 INFO:teuthology.orchestra.run.smithi112.stdout:1061 trunc from 0xedf6026 to 0x83b7d78 2024-09-04T21:54:09.936 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:09.936 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:09.945 INFO:teuthology.orchestra.run.smithi112.stdout:1063 write 0xc341fe6 thru 0xc34e2e3 (0xc2fe bytes) 2024-09-04T21:54:09.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:09.953 INFO:teuthology.orchestra.run.smithi112.stdout:1064 read 0x7c40c73 thru 0x7c43803 (0x2b91 bytes) 2024-09-04T21:54:09.953 INFO:teuthology.orchestra.run.smithi112.stdout:1067 write 0x58af234 thru 0x58bee3a (0xfc07 bytes) 2024-09-04T21:54:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:1068 punch from 0x3ecd6d9 to 0x3edc215, (0xeb3c bytes) 2024-09-04T21:54:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:09.972 INFO:teuthology.orchestra.run.smithi112.stdout:1070 write 0xd4802ce thru 0xd48b056 (0xad89 bytes) 2024-09-04T21:54:09.973 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:09.972+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:1071 read 0x787eecb thru 0x788064f (0x1785 bytes) 2024-09-04T21:54:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:1072 write 0x916e63a thru 0x9170b16 (0x24dd bytes) 2024-09-04T21:54:09.984 INFO:teuthology.orchestra.run.smithi112.stdout:1073 write 0xd2da8e6 thru 0xd2db68f (0xdaa bytes) 2024-09-04T21:54:09.990 INFO:teuthology.orchestra.run.smithi112.stdout:1076 read 0x55c451d thru 0x55cb2b0 (0x6d94 bytes) 2024-09-04T21:54:09.990 INFO:teuthology.orchestra.run.smithi112.stdout:1077 punch from 0xcc32f59 to 0xcc37ac4, (0x4b6b bytes) 2024-09-04T21:54:09.990 INFO:teuthology.orchestra.run.smithi112.stdout:1079 read 0x8c9b24d thru 0x8c9b619 (0x3cd bytes) 2024-09-04T21:54:09.990 INFO:teuthology.orchestra.run.smithi112.stdout:1080 write 0x23fd2fc thru 0x23fd354 (0x59 bytes) 2024-09-04T21:54:09.999 INFO:teuthology.orchestra.run.smithi112.stdout:1081 read 0xac72012 thru 0xac7273a (0x729 bytes) 2024-09-04T21:54:09.999 INFO:teuthology.orchestra.run.smithi112.stdout:1083 read 0x97684b0 thru 0x976a0ab (0x1bfc bytes) 2024-09-04T21:54:09.999 INFO:teuthology.orchestra.run.smithi112.stdout:1084 writesame 0x382387 thru 0x383f98 data_size 0x1(0x1c12 bytes) 2024-09-04T21:54:10.019 INFO:teuthology.orchestra.run.smithi112.stdout:1086 write 0x93233e3 thru 0x9329f5b (0x6b79 bytes) 2024-09-04T21:54:10.034 INFO:teuthology.orchestra.run.smithi112.stdout:1087 trunc from 0xd48b057 to 0x77b0a60 2024-09-04T21:54:10.060 INFO:teuthology.orchestra.run.smithi112.stdout:1088 trunc from 0x77b0a60 to 0x88f1a41 2024-09-04T21:54:10.074 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:10.072+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:10.082 INFO:teuthology.orchestra.run.smithi112.stdout:1092 writesame 0xac5d805 thru 0xac5f327 data_size 0x1(0x1b23 bytes) 2024-09-04T21:54:10.082 INFO:teuthology.orchestra.run.smithi112.stdout:1093 punch from 0x72e4020 to 0x72eac6e, (0x6c4e bytes) 2024-09-04T21:54:10.082 INFO:teuthology.orchestra.run.smithi112.stdout:1095 punch from 0x3b64693 to 0x3b6dfe4, (0x9951 bytes) 2024-09-04T21:54:10.099 INFO:teuthology.orchestra.run.smithi112.stdout:1097 trunc from 0xac5f328 to 0x24e596 2024-09-04T21:54:10.120 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:10.120 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:10.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:10.179 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:10.180+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:10.225 INFO:teuthology.orchestra.run.smithi112.stdout:1098 writesame 0xd563494 thru 0xd565677 data_size 0xc(0x21e4 bytes) 2024-09-04T21:54:10.225 INFO:teuthology.orchestra.run.smithi112.stdout:1099 read 0x1c59385 thru 0x1c5c0cf (0x2d4b bytes) 2024-09-04T21:54:10.225 INFO:teuthology.orchestra.run.smithi112.stdout:1101 punch from 0x115de03 to 0x116b10b, (0xd308 bytes) 2024-09-04T21:54:10.225 INFO:teuthology.orchestra.run.smithi112.stdout:1102 read 0x9b5493d thru 0x9b55b9d (0x1261 bytes) 2024-09-04T21:54:10.225 INFO:teuthology.orchestra.run.smithi112.stdout:1103 writesame 0x410b552 thru 0x4112a8e data_size 0x1(0x753d bytes) 2024-09-04T21:54:10.265 INFO:teuthology.orchestra.run.smithi112.stdout:1104 read 0x78e44c thru 0x790dc8 (0x297d bytes) 2024-09-04T21:54:10.266 INFO:teuthology.orchestra.run.smithi112.stdout:1107 punch from 0xa6e8d1d to 0xa6f38e5, (0xabc8 bytes) 2024-09-04T21:54:10.266 INFO:teuthology.orchestra.run.smithi112.stdout:1108 read 0xf61adf thru 0xf6e7c8 (0xccea bytes) 2024-09-04T21:54:10.266 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.266 INFO:teuthology.orchestra.run.smithi112.stdout:1110 punch from 0xb208e0f to 0xb20e8f5, (0x5ae6 bytes) 2024-09-04T21:54:10.266 INFO:teuthology.orchestra.run.smithi112.stdout:1111 write 0xd4a660f thru 0xd4ab171 (0x4b63 bytes) 2024-09-04T21:54:10.270 INFO:teuthology.orchestra.run.smithi112.stdout:1113 punch from 0xc7280c9 to 0xc729268, (0x119f bytes) 2024-09-04T21:54:10.270 INFO:teuthology.orchestra.run.smithi112.stdout:1114 writesame 0x14a3cef thru 0x14ac241 data_size 0x1f(0x8553 bytes) 2024-09-04T21:54:10.274 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.275 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:10.275+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:10.284 INFO:teuthology.orchestra.run.smithi112.stdout:1116 trunc from 0xd565678 to 0x6a4a97b 2024-09-04T21:54:10.303 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.303 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.303 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.320 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:10.320 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:10.322 INFO:teuthology.orchestra.run.smithi112.stdout:1120 write 0xd66c3d5 thru 0xd6729af (0x65db bytes) 2024-09-04T21:54:10.329 INFO:teuthology.orchestra.run.smithi112.stdout:1121 read 0x2da1d4d thru 0x2da6db1 (0x5065 bytes) 2024-09-04T21:54:10.329 INFO:teuthology.orchestra.run.smithi112.stdout:1123 write 0xabe5081 thru 0xabecf05 (0x7e85 bytes) 2024-09-04T21:54:10.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:10.336 INFO:teuthology.orchestra.run.smithi112.stdout:1124 write 0x2e65e6 thru 0x2ece3d (0x6858 bytes) 2024-09-04T21:54:10.339 INFO:teuthology.orchestra.run.smithi112.stdout:1128 read 0x7e388ad thru 0x7e4855c (0xfcb0 bytes) 2024-09-04T21:54:10.339 INFO:teuthology.orchestra.run.smithi112.stdout:1129 write 0xba1d330 thru 0xba240c8 (0x6d99 bytes) 2024-09-04T21:54:10.346 INFO:teuthology.orchestra.run.smithi112.stdout:1132 punch from 0xa32515c to 0xa331936, (0xc7da bytes) 2024-09-04T21:54:10.353 INFO:teuthology.orchestra.run.smithi112.stdout:1133 trunc from 0xd6729b0 to 0x863332e 2024-09-04T21:54:10.373 INFO:teuthology.orchestra.run.smithi112.stdout:1134 read 0xdc9589 thru 0xdd0ca9 (0x7721 bytes) 2024-09-04T21:54:10.373 INFO:teuthology.orchestra.run.smithi112.stdout:1135 write 0x51e7a30 thru 0x51ec0e1 (0x46b2 bytes) 2024-09-04T21:54:10.375 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:10.375+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:10.390 INFO:teuthology.orchestra.run.smithi112.stdout:1136 writesame 0xcd67f15 thru 0xcd6a779 data_size 0x9(0x2865 bytes) 2024-09-04T21:54:10.390 INFO:teuthology.orchestra.run.smithi112.stdout:1138 writesame 0x980c2f9 thru 0x980f5c3 data_size 0x1(0x32cb bytes) 2024-09-04T21:54:10.405 INFO:teuthology.orchestra.run.smithi112.stdout:1139 punch from 0x7910b0f to 0x792045f, (0xf950 bytes) 2024-09-04T21:54:10.405 INFO:teuthology.orchestra.run.smithi112.stdout:1140 write 0x4f69fd2 thru 0x4f73ecc (0x9efb bytes) 2024-09-04T21:54:10.411 INFO:teuthology.orchestra.run.smithi112.stdout:1141 writesame 0x48c9981 thru 0x48d5769 data_size 0x1(0xbde9 bytes) 2024-09-04T21:54:10.443 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:54:10.447 INFO:teuthology.orchestra.run.smithi112.stdout:1143 read 0xcb99a59 thru 0xcba4fe4 (0xb58c bytes) 2024-09-04T21:54:10.447 INFO:teuthology.orchestra.run.smithi112.stdout:1144 read 0x404b4fe thru 0x404c213 (0xd16 bytes) 2024-09-04T21:54:10.452 INFO:teuthology.orchestra.run.smithi112.stdout:1145 writesame 0xaf34132 thru 0xaf3cdeb data_size 0x1(0x8cba bytes) 2024-09-04T21:54:10.479 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:10.479 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:10.479 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:10.476+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:10.485 INFO:teuthology.orchestra.run.smithi112.stdout:1146 trunc from 0xcd6a77a to 0x82126bd 2024-09-04T21:54:10.498 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:10.518 INFO:teuthology.orchestra.run.smithi112.stdout:1147 write 0xe44d9d8 thru 0xe44ec77 (0x12a0 bytes) 2024-09-04T21:54:10.524 INFO:teuthology.orchestra.run.smithi112.stdout:1148 punch from 0x9a48342 to 0x9a4ec71, (0x692f bytes) 2024-09-04T21:54:10.524 INFO:teuthology.orchestra.run.smithi112.stdout:1149 writesame 0xc44f197 thru 0xc453168 data_size 0x15(0x3fd2 bytes) 2024-09-04T21:54:10.530 INFO:teuthology.orchestra.run.smithi112.stdout:1150 write 0x54e8065 thru 0x54f1c0f (0x9bab bytes) 2024-09-04T21:54:10.556 INFO:teuthology.orchestra.run.smithi112.stdout:1152 trunc from 0xe44ec78 to 0x1ff0932 2024-09-04T21:54:10.573 INFO:teuthology.orchestra.run.smithi112.stdout:1153 read 0x14c629e thru 0x14d4e43 (0xeba6 bytes) 2024-09-04T21:54:10.577 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:10.576+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:10.592 INFO:teuthology.orchestra.run.smithi112.stdout:1154 trunc from 0x1ff0932 to 0xe152376 2024-09-04T21:54:10.611 INFO:teuthology.orchestra.run.smithi112.stdout:1156 trunc from 0xe152376 to 0x5b741a7 2024-09-04T21:54:10.619 INFO:teuthology.orchestra.run.smithi112.stdout:1157 writesame 0xdef314 thru 0xdf38cc data_size 0x1(0x45b9 bytes) 2024-09-04T21:54:10.631 INFO:teuthology.orchestra.run.smithi112.stdout:1158 punch from 0x3d51636 to 0x3d549a3, (0x336d bytes) 2024-09-04T21:54:10.644 INFO:teuthology.orchestra.run.smithi112.stdout:1161 write 0x85afcfb thru 0x85b58ba (0x5bc0 bytes) 2024-09-04T21:54:10.650 INFO:teuthology.orchestra.run.smithi112.stdout:1162 punch from 0x74cd07f to 0x74d35a6, (0x6527 bytes) 2024-09-04T21:54:10.650 INFO:teuthology.orchestra.run.smithi112.stdout:1163 read 0x3699cef thru 0x36a348c (0x979e bytes) 2024-09-04T21:54:10.657 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:10.657 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:10.659 INFO:teuthology.orchestra.run.smithi112.stdout:1165 writesame 0x8f2b8b3 thru 0x8f2eb6f data_size 0x1(0x32bd bytes) 2024-09-04T21:54:10.668 INFO:teuthology.orchestra.run.smithi112.stdout:1169 trunc from 0x8f2eb70 to 0x3d3a565 2024-09-04T21:54:10.676 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:10.678 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:10.679+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:10.684 INFO:teuthology.orchestra.run.smithi112.stdout:1170 read 0x1d00a31 thru 0x1d04bfe (0x41ce bytes) 2024-09-04T21:54:10.685 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.694 INFO:teuthology.orchestra.run.smithi112.stdout:1172 write 0x4f997a1 thru 0x4fa74ba (0xdd1a bytes) 2024-09-04T21:54:10.718 INFO:teuthology.orchestra.run.smithi112.stdout:1173 trunc from 0x4fa74bb to 0xe978710 2024-09-04T21:54:10.726 INFO:teuthology.orchestra.run.smithi112.stdout:1174 writesame 0x4aa76df thru 0x4aad821 data_size 0x1(0x6143 bytes) 2024-09-04T21:54:10.744 INFO:teuthology.orchestra.run.smithi112.stdout:1175 read 0xbaeb95d thru 0xbaf5478 (0x9b1c bytes) 2024-09-04T21:54:10.744 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.745 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.745 INFO:teuthology.orchestra.run.smithi112.stdout:1178 write 0x429fac7 thru 0x42a808b (0x85c5 bytes) 2024-09-04T21:54:10.752 INFO:teuthology.orchestra.run.smithi112.stdout:1179 writesame 0xd33e8a7 thru 0xd342e8b data_size 0x1(0x45e5 bytes) 2024-09-04T21:54:10.768 INFO:teuthology.orchestra.run.smithi112.stdout:1180 read 0x5829e7f thru 0x5833f5b (0xa0dd bytes) 2024-09-04T21:54:10.769 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.769 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.769 INFO:teuthology.orchestra.run.smithi112.stdout:1183 read 0x9bc5713 thru 0x9bce2b4 (0x8ba2 bytes) 2024-09-04T21:54:10.779 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:10.780+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:10.788 INFO:teuthology.orchestra.run.smithi112.stdout:1184 trunc from 0xe978710 to 0x93e148 2024-09-04T21:54:10.809 INFO:teuthology.orchestra.run.smithi112.stdout:1185 read 0x10be88 thru 0x116984 (0xaafd bytes) 2024-09-04T21:54:10.814 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:10.815 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:10.837 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:10.838 INFO:teuthology.orchestra.run.smithi112.stdout:1186 write 0xb107fcc thru 0xb1167a1 (0xe7d6 bytes) 2024-09-04T21:54:10.843 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.843 INFO:teuthology.orchestra.run.smithi112.stdout:1190 read 0x2888981 thru 0x2888ee3 (0x563 bytes) 2024-09-04T21:54:10.844 INFO:teuthology.orchestra.run.smithi112.stdout:1191 writesame 0x902b4fb thru 0x9035420 data_size 0x2(0x9f26 bytes) 2024-09-04T21:54:10.858 INFO:teuthology.orchestra.run.smithi112.stdout:1192 punch from 0x4c2c952 to 0x4c31685, (0x4d33 bytes) 2024-09-04T21:54:10.873 INFO:teuthology.orchestra.run.smithi112.stdout:1194 writesame 0xd946387 thru 0xd94b610 data_size 0x5(0x528a bytes) 2024-09-04T21:54:10.873 INFO:teuthology.orchestra.run.smithi112.stdout:1195 write 0x3e377a thru 0x3ebd04 (0x858b bytes) 2024-09-04T21:54:10.880 INFO:teuthology.orchestra.run.smithi112.stdout:1196 read 0x7163a7f thru 0x71652a0 (0x1822 bytes) 2024-09-04T21:54:10.880 INFO:teuthology.orchestra.run.smithi112.stdout:1197 punch from 0x8674175 to 0x8678400, (0x428b bytes) 2024-09-04T21:54:10.880 INFO:teuthology.orchestra.run.smithi112.stdout:1201 write 0x4984367 thru 0x498cfa3 (0x8c3d bytes) 2024-09-04T21:54:10.881 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:10.879+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:10.887 INFO:teuthology.orchestra.run.smithi112.stdout:1203 read 0xd2be4d6 thru 0xd2c1d54 (0x387f bytes) 2024-09-04T21:54:10.888 INFO:teuthology.orchestra.run.smithi112.stdout:1204 writesame 0x8439acd thru 0x843a1e5 data_size 0x1(0x719 bytes) 2024-09-04T21:54:10.894 INFO:teuthology.orchestra.run.smithi112.stdout:1205 writesame 0x5d4890b thru 0x5d4bf13 data_size 0x1d(0x3609 bytes) 2024-09-04T21:54:10.901 INFO:teuthology.orchestra.run.smithi112.stdout:1206 write 0x671a676 thru 0x6721b3b (0x74c6 bytes) 2024-09-04T21:54:10.909 INFO:teuthology.orchestra.run.smithi112.stdout:1207 write 0x2303535 thru 0x230e7e0 (0xb2ac bytes) 2024-09-04T21:54:10.915 INFO:teuthology.orchestra.run.smithi112.stdout:1208 punch from 0xbc74766 to 0xbc841da, (0xfa74 bytes) 2024-09-04T21:54:10.915 INFO:teuthology.orchestra.run.smithi112.stdout:1209 read 0x669742f thru 0x669b647 (0x4219 bytes) 2024-09-04T21:54:10.915 INFO:teuthology.orchestra.run.smithi112.stdout:1210 punch from 0xb691d78 to 0xb69c5b4, (0xa83c bytes) 2024-09-04T21:54:10.918 INFO:teuthology.orchestra.run.smithi112.stdout:1212 trunc from 0xd94b611 to 0xc149c57 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1213 punch from 0x5e08896 to 0x5e0a0c7, (0x1831 bytes) 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1217 read 0x36776e0 thru 0x3679255 (0x1b76 bytes) 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1218 read 0x889a5fd thru 0x88a20bd (0x7ac1 bytes) 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1219 punch from 0x489e290 to 0x48a60d7, (0x7e47 bytes) 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1221 punch from 0x92c9494 to 0x92d6537, (0xd0a3 bytes) 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1223 read 0x5a7c0a6 thru 0x5a85176 (0x90d1 bytes) 2024-09-04T21:54:10.930 INFO:teuthology.orchestra.run.smithi112.stdout:1224 write 0x1e2fac6 thru 0x1e33c67 (0x41a2 bytes) 2024-09-04T21:54:10.937 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.937 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:10.937 INFO:teuthology.orchestra.run.smithi112.stdout:1228 write 0xb7e6f9c thru 0xb7ea139 (0x319e bytes) 2024-09-04T21:54:10.944 INFO:teuthology.orchestra.run.smithi112.stdout:1230 read 0x964206e thru 0x964da5a (0xb9ed bytes) 2024-09-04T21:54:10.944 INFO:teuthology.orchestra.run.smithi112.stdout:1233 writesame 0xb0d555b thru 0xb0da891 data_size 0x9(0x5337 bytes) 2024-09-04T21:54:10.952 INFO:teuthology.orchestra.run.smithi112.stdout:1234 writesame 0x215df3b thru 0x216ab74 data_size 0x1(0xcc3a bytes) 2024-09-04T21:54:10.966 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:10.967 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:10.981 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:10.980+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:10.984 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:11.009 INFO:teuthology.orchestra.run.smithi112.stdout:1235 write 0x627bf24 thru 0x628af9d (0xf07a bytes) 2024-09-04T21:54:11.016 INFO:teuthology.orchestra.run.smithi112.stdout:1237 write 0x8086ebc thru 0x808da7a (0x6bbf bytes) 2024-09-04T21:54:11.022 INFO:teuthology.orchestra.run.smithi112.stdout:1238 writesame 0x8d57a4b thru 0x8d59cb8 data_size 0x27(0x226e bytes) 2024-09-04T21:54:11.036 INFO:teuthology.orchestra.run.smithi112.stdout:1239 write 0xc25907f thru 0xc25b18b (0x210d bytes) 2024-09-04T21:54:11.040 INFO:teuthology.orchestra.run.smithi112.stdout:1240 punch from 0x39cc10c to 0x39ce041, (0x1f35 bytes) 2024-09-04T21:54:11.040 INFO:teuthology.orchestra.run.smithi112.stdout:1243 write 0x531548e thru 0x5322c5f (0xd7d2 bytes) 2024-09-04T21:54:11.047 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.047 INFO:teuthology.orchestra.run.smithi112.stdout:1246 write 0x4f899aa thru 0x4f8dbdb (0x4232 bytes) 2024-09-04T21:54:11.052 INFO:teuthology.orchestra.run.smithi112.stdout:1247 read 0xbde86ef thru 0xbdf469a (0xbfac bytes) 2024-09-04T21:54:11.052 INFO:teuthology.orchestra.run.smithi112.stdout:1248 punch from 0xaad916b to 0xaae0677, (0x750c bytes) 2024-09-04T21:54:11.055 INFO:teuthology.orchestra.run.smithi112.stdout:1249 trunc from 0xc25b18c to 0xafa38cf 2024-09-04T21:54:11.076 INFO:teuthology.orchestra.run.smithi112.stdout:1251 punch from 0xacfc333 to 0xacfdafb, (0x17c8 bytes) 2024-09-04T21:54:11.082 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:11.079+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:11.086 INFO:teuthology.orchestra.run.smithi112.stdout:1252 trunc from 0xafa38cf to 0x60d8ff9 2024-09-04T21:54:11.113 INFO:teuthology.orchestra.run.smithi112.stdout:1253 read 0x2d77ff7 thru 0x2d8763c (0xf646 bytes) 2024-09-04T21:54:11.114 INFO:teuthology.orchestra.run.smithi112.stdout:1254 write 0x51ddacf thru 0x51dfe54 (0x2386 bytes) 2024-09-04T21:54:11.120 INFO:teuthology.orchestra.run.smithi112.stdout:1255 trunc from 0x60d8ff9 to 0x5fc7cb8 2024-09-04T21:54:11.124 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:11.124 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:11.139 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:11.166 INFO:teuthology.orchestra.run.smithi112.stdout:1256 write 0xa751d38 thru 0xa75ca61 (0xad2a bytes) 2024-09-04T21:54:11.183 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:11.179+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:11.185 INFO:teuthology.orchestra.run.smithi112.stdout:1257 write 0xa8fca44 thru 0xa900dd2 (0x438f bytes) 2024-09-04T21:54:11.193 INFO:teuthology.orchestra.run.smithi112.stdout:1258 write 0x1b9ed5f thru 0x1ba60ba (0x735c bytes) 2024-09-04T21:54:11.209 INFO:teuthology.orchestra.run.smithi112.stdout:1259 trunc from 0xa900dd3 to 0xd0c5b5d 2024-09-04T21:54:11.222 INFO:teuthology.orchestra.run.smithi112.stdout:1261 write 0x9bb0039 thru 0x9bb7405 (0x73cd bytes) 2024-09-04T21:54:11.229 INFO:teuthology.orchestra.run.smithi112.stdout:1262 write 0xc3d801a thru 0xc3db203 (0x31ea bytes) 2024-09-04T21:54:11.236 INFO:teuthology.orchestra.run.smithi112.stdout:1265 writesame 0x987c36e thru 0x9885bfc data_size 0x5(0x988f bytes) 2024-09-04T21:54:11.264 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:11.265 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:11.271 INFO:teuthology.orchestra.run.smithi112.stdout:1266 trunc from 0xd0c5b5d to 0x3ee5576 2024-09-04T21:54:11.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:11.279 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:11.295 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.295 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:11.295+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:11.320 INFO:teuthology.orchestra.run.smithi112.stdout:1268 writesame 0xce47cf9 thru 0xce4fb84 data_size 0x5b(0x7e8c bytes) 2024-09-04T21:54:11.339 INFO:teuthology.orchestra.run.smithi112.stdout:1269 trunc from 0xce4fb85 to 0x17ec9ed 2024-09-04T21:54:11.374 INFO:teuthology.orchestra.run.smithi112.stdout:1270 writesame 0x79040a5 thru 0x7904bc8 data_size 0x1f(0xb24 bytes) 2024-09-04T21:54:11.384 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:11.383+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:11.387 INFO:teuthology.orchestra.run.smithi112.stdout:1271 write 0x9d23bb6 thru 0x9d24de3 (0x122e bytes) 2024-09-04T21:54:11.393 INFO:teuthology.orchestra.run.smithi112.stdout:1272 punch from 0x7d832e8 to 0x7d8f060, (0xbd78 bytes) 2024-09-04T21:54:11.393 INFO:teuthology.orchestra.run.smithi112.stdout:1273 write 0x2213c83 thru 0x22238b8 (0xfc36 bytes) 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:1274 read 0x1d01549 thru 0x1d0e31e (0xcdd6 bytes) 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:1275 read 0x7274c20 thru 0x727788a (0x2c6b bytes) 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:1276 read 0x2273e76 thru 0x2275cb2 (0x1e3d bytes) 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:1277 punch from 0x9ac1d1f to 0x9ace3bd, (0xc69e bytes) 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.401 INFO:teuthology.orchestra.run.smithi112.stdout:1281 write 0x947db2 thru 0x952ade (0xad2d bytes) 2024-09-04T21:54:11.406 INFO:teuthology.orchestra.run.smithi112.stdout:1282 read 0x81838b1 thru 0x818be14 (0x8564 bytes) 2024-09-04T21:54:11.406 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.406 INFO:teuthology.orchestra.run.smithi112.stdout:1284 read 0xe459aa thru 0xe4812d (0x2784 bytes) 2024-09-04T21:54:11.406 INFO:teuthology.orchestra.run.smithi112.stdout:1285 read 0x8e25d5 thru 0x8e50c7 (0x2af3 bytes) 2024-09-04T21:54:11.408 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:11.408 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:11.413 INFO:teuthology.orchestra.run.smithi112.stdout:1286 trunc from 0x9d24de4 to 0x4fe4d7b 2024-09-04T21:54:11.423 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:11.442 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:11.442 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:11.444 INFO:teuthology.orchestra.run.smithi112.stdout:1287 writesame 0xdd0e651 thru 0xdd177fe data_size 0x1d(0x91ae bytes) 2024-09-04T21:54:11.445 INFO:teuthology.orchestra.run.smithi112.stdout:1289 write 0x51221b9 thru 0x512ed06 (0xcb4e bytes) 2024-09-04T21:54:11.449 INFO:teuthology.orchestra.run.smithi112.stdout:1290 trunc from 0xdd177ff to 0xea57e41 2024-09-04T21:54:11.461 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:11.461 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:11.473 INFO:teuthology.orchestra.run.smithi112.stdout:1292 trunc from 0xea57e41 to 0x2958b05 2024-09-04T21:54:11.480 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:11.480 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:11.485 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:11.483+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:11.497 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:11.497 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:11.507 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.507 INFO:teuthology.orchestra.run.smithi112.stdout:1295 read 0x100c061 thru 0x1019a00 (0xd9a0 bytes) 2024-09-04T21:54:11.507 INFO:teuthology.orchestra.run.smithi112.stdout:1297 punch from 0x2809830 to 0x2818d80, (0xf550 bytes) 2024-09-04T21:54:11.514 INFO:teuthology.orchestra.run.smithi112.stdout:1298 trunc from 0x2958b05 to 0x7894352 2024-09-04T21:54:11.516 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:11.516 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:11.521 INFO:teuthology.orchestra.run.smithi112.stdout:1300 punch from 0xf40f0 to 0x102e36, (0xed46 bytes) 2024-09-04T21:54:11.527 INFO:teuthology.orchestra.run.smithi112.stdout:1301 write 0x7eb4175 thru 0x7eb5542 (0x13ce bytes) 2024-09-04T21:54:11.531 INFO:teuthology.orchestra.run.smithi112.stdout:1302 read 0x32a9eb4 thru 0x32b2319 (0x8466 bytes) 2024-09-04T21:54:11.531 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.531 INFO:teuthology.orchestra.run.smithi112.stdout:1304 read 0x10705b2 thru 0x1070b64 (0x5b3 bytes) 2024-09-04T21:54:11.536 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:11.536 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:11.539 INFO:teuthology.orchestra.run.smithi112.stdout:1305 trunc from 0x7eb5543 to 0xdcdafbb 2024-09-04T21:54:11.545 INFO:teuthology.orchestra.run.smithi112.stdout:1306 write 0xa5f3210 thru 0xa5f9171 (0x5f62 bytes) 2024-09-04T21:54:11.556 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:11.556 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:11.557 INFO:teuthology.orchestra.run.smithi112.stdout:1307 trunc from 0xdcdafbb to 0x99fb1dd 2024-09-04T21:54:11.557 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:11.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:11.568 INFO:teuthology.orchestra.run.smithi112.stdout:1308 read 0x71bca47 thru 0x71c881e (0xbdd8 bytes) 2024-09-04T21:54:11.568 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.568 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.569 INFO:teuthology.orchestra.run.smithi112.stdout:1315 punch from 0x6da1b40 to 0x6da9f06, (0x83c6 bytes) 2024-09-04T21:54:11.569 INFO:teuthology.orchestra.run.smithi112.stdout:1317 write 0x4e42c97 thru 0x4e46824 (0x3b8e bytes) 2024-09-04T21:54:11.571 INFO:teuthology.orchestra.run.smithi112.stdout:1318 punch from 0x88a324 to 0x891016, (0x6cf2 bytes) 2024-09-04T21:54:11.584 INFO:teuthology.orchestra.run.smithi112.stdout:1319 trunc from 0x99fb1dd to 0x2ba8fd1 2024-09-04T21:54:11.586 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:11.583+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:11.598 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:11.598 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:11.598 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:11.598 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:11.608 INFO:teuthology.orchestra.run.smithi112.stdout:1320 trunc from 0x2ba8fd1 to 0xa2b1cd0 2024-09-04T21:54:11.612 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:11.627 INFO:teuthology.orchestra.run.smithi112.stdout:1321 trunc from 0xa2b1cd0 to 0x42cb3cb 2024-09-04T21:54:11.636 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.636 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.636 INFO:teuthology.orchestra.run.smithi112.stdout:1325 read 0x186c830 thru 0x187362a (0x6dfb bytes) 2024-09-04T21:54:11.656 INFO:teuthology.orchestra.run.smithi112.stdout:1326 writesame 0x97dd72b thru 0x97e1664 data_size 0x1(0x3f3a bytes) 2024-09-04T21:54:11.656 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:11.656 INFO:teuthology.orchestra.run.smithi112.stdout:1328 writesame 0x22d8785 thru 0x22ddc37 data_size 0x1(0x54b3 bytes) 2024-09-04T21:54:11.665 INFO:teuthology.orchestra.run.smithi112.stdout:1329 read 0x4f6191a thru 0x4f64242 (0x2929 bytes) 2024-09-04T21:54:11.665 INFO:teuthology.orchestra.run.smithi112.stdout:1330 read 0x31a4d2 thru 0x325727 (0xb256 bytes) 2024-09-04T21:54:11.682 INFO:teuthology.orchestra.run.smithi112.stdout:1331 writesame 0xbe1d9ea thru 0xbe1e2ff data_size 0x1(0x916 bytes) 2024-09-04T21:54:11.682 INFO:teuthology.orchestra.run.smithi112.stdout:1333 write 0x2ded62b thru 0x2df6ad7 (0x94ad bytes) 2024-09-04T21:54:11.686 INFO:teuthology.orchestra.run.smithi112.stdout:1334 clone 6 order 24 su 131072 sc 10 2024-09-04T21:54:11.687 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:11.683+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:11.735 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:11.735 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:11.735 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:11.736 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:11.736 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:11.749 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:11.787 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:11.783+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:11.873 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.068567544, 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068597409999999999, 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.874 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4208", 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/1349276815", 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804258+0000", 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804261+0000", 2024-09-04T21:54:11.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0230000000000001e-06 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804276+0000", 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4304000000000001e-05 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804309+0000", 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3463000000000001e-05 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804328+0000", 2024-09-04T21:54:11.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9307e-05 2024-09-04T21:54:11.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804489+0000", 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000160672 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.804499+0000", 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0283e-05 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.807660+0000", 2024-09-04T21:54:11.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003160969 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.807887+0000", 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022673800000000001 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.808103+0000", 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021598900000000001 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.809574+0000", 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014714940000000001 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.810928+0000", 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001353344 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.811009+0000", 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1406e-05 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:46.811118+0000", 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108749 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 24.064210239000001, 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0075919359999999996, 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4208", 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/1349276815", 2024-09-04T21:54:11.883 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808616+0000", 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808619+0000", 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808652+0000", 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2842999999999997e-05 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808686+0000", 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4205999999999997e-05 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808703+0000", 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7597000000000001e-05 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808872+0000", 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016823599999999999 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:11.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.808889+0000", 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7716000000000001e-05 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813206+0000", 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043164680000000004 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813380+0000", 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017376199999999999 2024-09-04T21:54:11.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1", 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.813595+0000", 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021576899999999999 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.815122+0000", 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015264339999999999 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816053+0000", 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00093072599999999999 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816141+0000", 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8462999999999996e-05 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:53:47.816207+0000", 2024-09-04T21:54:11.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6429999999999999e-05 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1027 3.1c 3:3c9626f4:::rbd_data.11882e693cd0.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9661043280000001, 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056794660000000002, 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.891 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1027 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906721+0000", 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906724+0000", 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4739999999999999e-06 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906736+0000", 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2e-05 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906768+0000", 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2106000000000001e-05 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906789+0000", 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1313000000000001e-05 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.906990+0000", 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020057300000000001 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907213+0000", 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022263099999999999 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.907521+0000", 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000308485 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.909663+0000", 2024-09-04T21:54:11.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021414559999999999 2024-09-04T21:54:11.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912156+0000", 2024-09-04T21:54:11.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002493033 2024-09-04T21:54:11.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912289+0000", 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013296900000000001 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912401+0000", 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112426 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.898 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1188 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2944860479999996, 2024-09-04T21:54:11.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054046620000000002, 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1188 2024-09-04T21:54:11.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578340+0000", 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1900000000000002e-07 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578348+0000", 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3560000000000004e-06 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578355+0000", 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.114e-06 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578359+0000", 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4660000000000001e-06 2024-09-04T21:54:11.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.579323+0000", 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096376800000000005 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.579621+0000", 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029837100000000002 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.580410+0000", 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078867200000000003 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583596+0000", 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031858160000000002 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583615+0000", 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9224000000000001e-05 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583678+0000", 2024-09-04T21:54:11.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2613999999999994e-05 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583744+0000", 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6642000000000003e-05 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1189 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2944579489999999, 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056526730000000004, 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1189 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578368+0000", 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6100000000000002e-07 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578370+0000", 2024-09-04T21:54:11.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2479999999999999e-06 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578375+0000", 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3819999999999997e-06 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.578380+0000", 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0509999999999996e-06 2024-09-04T21:54:11.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.581409+0000", 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030287550000000002 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.581628+0000", 2024-09-04T21:54:11.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021934499999999999 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.582249+0000", 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062055900000000002 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583761+0000", 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015120960000000001 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583845+0000", 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4357000000000006e-05 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.583945+0000", 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9550999999999996e-05 2024-09-04T21:54:11.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584021+0000", 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5968000000000002e-05 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1248 3.16 3:6ce50ae3:::rbd_data.1188e7c968d9.0000000000000087:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1953345790000007, 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013362881, 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1248 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677491+0000", 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677492+0000", 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.035e-06 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677497+0000", 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5050000000000001e-06 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677507+0000", 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8679999999999994e-06 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677514+0000", 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7049999999999999e-06 2024-09-04T21:54:11.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677595+0000", 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0308000000000004e-05 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.677748+0000", 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000153425 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690686+0000", 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.01293805 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690748+0000", 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2189999999999999e-05 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.690854+0000", 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105795 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1303 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0668728010000006, 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0064365610000000004, 2024-09-04T21:54:11.922 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1303 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805953+0000", 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805954+0000", 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.285e-06 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805960+0000", 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0460000000000002e-06 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805975+0000", 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4304999999999999e-05 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.805981+0000", 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5919999999999997e-06 2024-09-04T21:54:11.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806084+0000", 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103027 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806256+0000", 2024-09-04T21:54:11.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017127900000000001 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806853+0000", 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00059703999999999996 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.811952+0000", 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050998559999999998 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812130+0000", 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017728899999999999 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812269+0000", 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013945900000000001 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812390+0000", 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120383 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1304 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0668157310000002, 2024-09-04T21:54:11.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066469160000000001, 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1304 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806010+0000", 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806011+0000", 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4000000000000001e-07 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806014+0000", 2024-09-04T21:54:11.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.444e-06 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806030+0000", 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5480000000000001e-05 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806035+0000", 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0649999999999998e-06 2024-09-04T21:54:11.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.807688+0000", 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016533559999999999 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.807956+0000", 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000268182 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.809072+0000", 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001115659 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812090+0000", 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003018097 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812415+0000", 2024-09-04T21:54:11.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032458399999999999 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812546+0000", 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013182600000000001 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812657+0000", 2024-09-04T21:54:11.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011048300000000001 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1305 3.1d 3:be19795e:::rbd_object_map.1188e7c968d9:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:11.938 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0667634830000008, 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073242790000000004, 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1305 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806062+0000", 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806063+0000", 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4400000000000001e-07 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806065+0000", 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5450000000000002e-06 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806072+0000", 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.806075+0000", 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4960000000000001e-06 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.810223+0000", 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041482749999999999 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.810500+0000", 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027680199999999998 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.811129+0000", 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062907200000000005 2024-09-04T21:54:11.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.812801+0000", 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001671753 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813238+0000", 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043750000000000001 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813321+0000", 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2145999999999996e-05 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.813387+0000", 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6045999999999998e-05 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1334 3.c 3:303b0518:::rbd_data.1188e7c968d9.0000000000000101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0202318199999993, 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056592250000000004, 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.946 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1334 2024-09-04T21:54:11.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.947 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:11.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852594+0000", 2024-09-04T21:54:11.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852595+0000", 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.322e-06 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852602+0000", 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3119999999999999e-06 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852623+0000", 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1106000000000001e-05 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852636+0000", 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3692e-05 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852743+0000", 2024-09-04T21:54:11.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106747 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.852884+0000", 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000140755 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853074+0000", 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019004499999999999 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854595+0000", 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001521175 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858170+0000", 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035752470000000001 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858216+0000", 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5948000000000001e-05 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858253+0000", 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6875999999999997e-05 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7673 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735557+0000", 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.1372685169999999, 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005252278, 2024-09-04T21:54:11.953 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7673 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735557+0000", 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735557+0000", 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735558+0000", 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3300000000000002e-07 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735561+0000", 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.23e-06 2024-09-04T21:54:11.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735575+0000", 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4422e-05 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735581+0000", 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736854+0000", 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012722320000000001 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736966+0000", 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112237 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.737490+0000", 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00052420400000000001 2024-09-04T21:54:11.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739475+0000", 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019846130000000001 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740683+0000", 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012080019999999999 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740764+0000", 2024-09-04T21:54:11.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1564999999999996e-05 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740810+0000", 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5373000000000001e-05 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7674 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:11.959 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735590+0000", 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.1372354089999996, 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0058458, 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7674 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735590+0000", 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735590+0000", 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735591+0000", 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8700000000000001e-07 2024-09-04T21:54:11.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735595+0000", 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.912e-06 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735602+0000", 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1149999999999996e-06 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735606+0000", 2024-09-04T21:54:11.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0119999999999999e-06 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738398+0000", 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002792414 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738503+0000", 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105046 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738961+0000", 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045731800000000001 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739909+0000", 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094838400000000003 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741310+0000", 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014005860000000001 2024-09-04T21:54:11.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741379+0000", 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9648999999999997e-05 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741436+0000", 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6977e-05 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7678 3.c 3:30b5c6f9:::rbd_data.1188b1294ada.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.740524+0000", 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.1323017909999997, 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006006323, 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7678 2024-09-04T21:54:11.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740524+0000", 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740524+0000", 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.740526+0000", 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1440000000000001e-06 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741421+0000", 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000894892 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741435+0000", 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3991e-05 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741442+0000", 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2960000000000001e-06 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741519+0000", 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6983999999999993e-05 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741656+0000", 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000136468 2024-09-04T21:54:11.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741963+0000", 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030737899999999999 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.744947+0000", 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029836860000000002 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.746457+0000", 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00151012 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.746501+0000", 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3680999999999997e-05 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.746530+0000", 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9682000000000001e-05 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.971 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7867 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572350+0000", 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3004755110000001, 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0062892809999999999, 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.972 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7867 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572350+0000", 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572350+0000", 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572351+0000", 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8599999999999997e-07 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572355+0000", 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2869999999999998e-06 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572365+0000", 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2949999999999999e-06 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572370+0000", 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9540000000000003e-06 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572408+0000", 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8438e-05 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572475+0000", 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7426000000000001e-05 2024-09-04T21:54:11.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572739+0000", 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000263131 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.575040+0000", 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023015930000000002 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578550+0000", 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035097969999999998 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578597+0000", 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148000000000003e-05 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578640+0000", 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2425999999999997e-05 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.977 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7868 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572455+0000", 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3003711359999999, 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0064475269999999998, 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7868 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572455+0000", 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572455+0000", 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572455+0000", 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2399999999999997e-07 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572460+0000", 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7940000000000002e-06 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572469+0000", 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.405e-06 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572484+0000", 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.483e-05 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.573002+0000", 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000518598 2024-09-04T21:54:11.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.573058+0000", 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6289999999999998e-05 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.573256+0000", 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019750200000000001 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576606+0000", 2024-09-04T21:54:11.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003350104 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578831+0000", 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022254280000000002 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578864+0000", 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2119999999999997e-05 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578902+0000", 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8631999999999999e-05 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7869 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572499+0000", 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3003264589999999, 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074274670000000001, 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7869 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572499+0000", 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572499+0000", 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572500+0000", 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.37e-07 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572508+0000", 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.225e-06 2024-09-04T21:54:11.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572515+0000", 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0539999999999997e-06 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572522+0000", 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0570000000000004e-06 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.573580+0000", 2024-09-04T21:54:11.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001057931 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.573710+0000", 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013013599999999999 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.574067+0000", 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035620600000000002 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576613+0000", 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025464390000000002 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579795+0000", 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031818419999999998 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579870+0000", 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5050999999999997e-05 2024-09-04T21:54:11.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579927+0000", 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6888999999999997e-05 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7870 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572533+0000", 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3002925589999998, 2024-09-04T21:54:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0075505659999999999, 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7870 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572533+0000", 2024-09-04T21:54:11.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572533+0000", 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572534+0000", 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.06e-07 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572537+0000", 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9739999999999998e-06 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572543+0000", 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5799999999999997e-06 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572549+0000", 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2889999999999997e-06 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.574477+0000", 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001928833 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.574525+0000", 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8019999999999998e-05 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.574746+0000", 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000220616 2024-09-04T21:54:11.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576620+0000", 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001873593 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579987+0000", 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033669949999999998 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:11.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.580030+0000", 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3470000000000002e-05 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.580084+0000", 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3690000000000003e-05 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7871 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:11.996 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572556+0000", 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3002701999999999, 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0083738900000000002, 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7871 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572556+0000", 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572556+0000", 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572556+0000", 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572562+0000", 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1480000000000003e-06 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572569+0000", 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5409999999999997e-06 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:11.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572574+0000", 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9429999999999999e-06 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.575090+0000", 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002516431 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.575217+0000", 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000126802 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.575595+0000", 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037764199999999997 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576623+0000", 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010286220000000001 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.580842+0000", 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042191260000000001 2024-09-04T21:54:12.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.580889+0000", 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6921e-05 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.580929+0000", 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0261e-05 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7872 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572581+0000", 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3002443719999999, 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0085113889999999994, 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.004 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7872 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572581+0000", 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572581+0000", 2024-09-04T21:54:12.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572582+0000", 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572585+0000", 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7829999999999999e-06 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572591+0000", 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5400000000000001e-06 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572596+0000", 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1680000000000003e-06 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576022+0000", 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003425343 2024-09-04T21:54:12.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576071+0000", 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9802999999999997e-05 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576280+0000", 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020856600000000001 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.577510+0000", 2024-09-04T21:54:12.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012296919999999999 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.580979+0000", 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003468839 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581029+0000", 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0519999999999997e-05 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581093+0000", 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3706000000000003e-05 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7873 3.5 3:a105420e:::rbd_object_map.1188b1294ada:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.572604+0000", 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.3002222240000001, 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0092477659999999993, 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.010 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7873 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572604+0000", 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572604+0000", 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572604+0000", 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3599999999999999e-07 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572607+0000", 2024-09-04T21:54:12.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.543e-06 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572613+0000", 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1959999999999996e-06 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.572618+0000", 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1549999999999998e-06 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576662+0000", 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004044382 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.576788+0000", 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012575299999999999 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.577144+0000", 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035567500000000001 2024-09-04T21:54:12.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.577860+0000", 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00071620899999999999 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581741+0000", 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038807960000000002 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581804+0000", 2024-09-04T21:54:12.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3039000000000004e-05 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581851+0000", 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7581999999999999e-05 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.016 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:12.018 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:12.018 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:12.015+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:12.040 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-04T21:54:12.040 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:54:12.040 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:54:12.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-04T21:54:12.118 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:12.115+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:12.143 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:12.144 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:12.144 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:12.144 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:12.158 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:12.220 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:12.219+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:12.282 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:12.282 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:12.282 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:12.282 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:12.282 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:12.297 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:12.342 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:12.342+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:12.368 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-04T21:54:12.367+0000 7f1d9e303640 -1 mon.a@0(leader).osd e34 definitely_dead 0 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1045 3.10 3:08ed8dc3:::rbd_data.11882e693cd0.000000000000002f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:12.422 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.507726644, 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054299810000000004, 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1045 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.423 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912477+0000", 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8369999999999999e-06 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912492+0000", 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.428e-05 2024-09-04T21:54:12.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912529+0000", 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7673000000000003e-05 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912547+0000", 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7689e-05 2024-09-04T21:54:12.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912710+0000", 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162526 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912993+0000", 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028371199999999998 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913324+0000", 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033046099999999999 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915482+0000", 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021584989999999999 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917665+0000", 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021827449999999998 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917778+0000", 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000113063 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917904+0000", 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012549600000000001 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.430 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1101 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9998706019999997, 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0067943780000000002, 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1101 2024-09-04T21:54:12.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420333+0000", 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6169999999999998e-06 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420347+0000", 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-05 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420381+0000", 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4122999999999998e-05 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420398+0000", 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6286999999999998e-05 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420568+0000", 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017088199999999999 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420589+0000", 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0513000000000002e-05 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424142+0000", 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035527359999999999 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424352+0000", 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021066300000000001 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424715+0000", 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000362233 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426277+0000", 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015626069999999999 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426969+0000", 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069220800000000004 2024-09-04T21:54:12.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427060+0000", 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1057000000000006e-05 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427124+0000", 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3522000000000006e-05 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.438 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1102 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_stripe_unit_count in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9928415590000004, 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026178130000000001, 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1102 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.439 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427361+0000", 2024-09-04T21:54:12.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.384e-06 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427371+0000", 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0086e-05 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427394+0000", 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2643e-05 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427406+0000", 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2247e-05 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427515+0000", 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010909899999999999 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427646+0000", 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013051299999999999 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.428016+0000", 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037050700000000001 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429054+0000", 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010376230000000001 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429807+0000", 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075308500000000002 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429894+0000", 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7216e-05 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429976+0000", 2024-09-04T21:54:12.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2410000000000005e-05 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1116 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [watch watch cookie 139952113516272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9727632360000005, 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029237209999999998, 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1116 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447439+0000", 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447449+0000", 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8239999999999995e-06 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447481+0000", 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1733000000000003e-05 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447497+0000", 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5653000000000001e-05 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447642+0000", 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014499599999999999 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447778+0000", 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013675199999999999 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.448006+0000", 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022790500000000001 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.449250+0000", 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001244172 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450163+0000", 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00091221100000000001 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450248+0000", 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5753000000000004e-05 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450361+0000", 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011235600000000001 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1192 3.a 3:52245ef8:::rbd_data.1188e7c968d9.00000000000001c6:head [set-alloc-hint object_size 524288 write_size 524288,write 131042~30] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:12.451 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8360188950000005, 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027730150000000002, 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1192 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584184+0000", 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8629999999999999e-06 2024-09-04T21:54:12.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584198+0000", 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3961999999999999e-05 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584231+0000", 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2660000000000002e-05 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584250+0000", 2024-09-04T21:54:12.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.908e-05 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584339+0000", 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9456000000000005e-05 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584499+0000", 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000159431 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584675+0000", 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017591899999999999 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586136+0000", 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014611050000000001 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586833+0000", 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069698700000000004 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586903+0000", 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0216000000000007e-05 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586954+0000", 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1335999999999998e-05 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1267 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.457 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.6909342899999995, 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026536139999999999, 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1267 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:12.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729268+0000", 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3800000000000001e-06 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729279+0000", 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0749e-05 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729309+0000", 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0029999999999999e-05 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729323+0000", 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.411e-05 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729437+0000", 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114115 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729610+0000", 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000172821 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729965+0000", 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035526999999999998 2024-09-04T21:54:12.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.730787+0000", 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082178999999999998 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731730+0000", 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094278199999999997 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731845+0000", 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114935 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731920+0000", 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4632e-05 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1279 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.463 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.6564715109999995, 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027977710000000001, 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1279 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763731+0000", 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.537e-06 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763743+0000", 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1426999999999999e-05 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763775+0000", 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1925999999999999e-05 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763789+0000", 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4756999999999999e-05 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763900+0000", 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011079800000000001 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.764069+0000", 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169307 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.764651+0000", 2024-09-04T21:54:12.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058184199999999995 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766213+0000", 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015619130000000001 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766364+0000", 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000150502 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766453+0000", 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8902999999999999e-05 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766526+0000", 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3858999999999999e-05 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.469 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1338 3.10 3:0a516185:::rbd_data.1188e7c968d9.000000000000010d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.5665656989999999, 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050744040000000002, 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.470 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1338 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853638+0000", 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853649+0000", 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1161999999999999e-05 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853682+0000", 2024-09-04T21:54:12.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3127000000000003e-05 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853697+0000", 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5318000000000001e-05 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853801+0000", 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010342500000000001 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853935+0000", 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000133771 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854097+0000", 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016247399999999999 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855226+0000", 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011287210000000001 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858515+0000", 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032894249999999999 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858634+0000", 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119226 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858709+0000", 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4468999999999995e-05 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1401 3.10 3:0966b599:::rbd_data.1188e7c968d9.0000000000000170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.476 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.4574554630000005, 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026384160000000002, 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1401 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962748+0000", 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:54:12.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962757+0000", 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1600000000000004e-06 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962781+0000", 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4426e-05 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962793+0000", 2024-09-04T21:54:12.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1955e-05 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962928+0000", 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135282 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963165+0000", 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023646100000000001 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963584+0000", 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041887000000000001 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965107+0000", 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015236970000000001 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965129+0000", 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.139e-05 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965258+0000", 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012891199999999999 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965383+0000", 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012537900000000001 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1612 3.10 3:0aa9c1cb:::rbd_data.1188e7c968d9.00000000000001a4:head [set-alloc-hint object_size 524288 write_size 524288,write 426878~31874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0763128000000002, 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003599587, 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1612 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343891+0000", 2024-09-04T21:54:12.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.23e-06 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343991+0000", 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010068000000000001 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344028+0000", 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6579000000000003e-05 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344049+0000", 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1002999999999999e-05 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344136+0000", 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7100000000000003e-05 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344241+0000", 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010485200000000001 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344342+0000", 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100721 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.345192+0000", 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085025300000000003 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347429+0000", 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022367620000000002 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347462+0000", 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2960999999999998e-05 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347487+0000", 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5446e-05 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1615 3.10 3:0aa9c1cb:::rbd_data.1188e7c968d9.00000000000001a4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:12.490 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.063500092, 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029358750000000001, 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1615 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.491 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356702+0000", 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1830000000000001e-06 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356711+0000", 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5469999999999997e-06 2024-09-04T21:54:12.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356769+0000", 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7896000000000003e-05 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356789+0000", 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0239999999999999e-05 2024-09-04T21:54:12.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356866+0000", 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6822000000000004e-05 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356941+0000", 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5223999999999997e-05 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.357037+0000", 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6063000000000005e-05 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.357780+0000", 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074262700000000004 2024-09-04T21:54:12.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359531+0000", 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001751072 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359588+0000", 2024-09-04T21:54:12.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6884e-05 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359636+0000", 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8316999999999998e-05 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7222 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [watch watch cookie 139952180803216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6030313310000004, 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031156790000000001, 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.499 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7222 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817171+0000", 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0600000000000002e-06 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817180+0000", 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3300000000000005e-06 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817213+0000", 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2750999999999999e-05 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817229+0000", 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5498e-05 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817353+0000", 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012423600000000001 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817584+0000", 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000230938 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817997+0000", 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041338900000000001 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.819260+0000", 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012629690000000001 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820086+0000", 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082551100000000002 2024-09-04T21:54:12.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820175+0000", 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104999999999995e-05 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820285+0000", 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109892 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7465 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1039991~8585] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0472689190000004, 2024-09-04T21:54:12.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030096060000000002, 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7465 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372934+0000", 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372947+0000", 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3232e-05 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372962+0000", 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5656000000000001e-05 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372970+0000", 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7500000000000003e-06 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373031+0000", 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1197000000000004e-05 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373143+0000", 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011143599999999999 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373258+0000", 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114842 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.374309+0000", 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010515570000000001 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375856+0000", 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001547185 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375909+0000", 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2335e-05 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375941+0000", 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2110000000000003e-05 2024-09-04T21:54:12.514 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7519 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9251712430000003, 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054280980000000001, 2024-09-04T21:54:12.515 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7519 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495033+0000", 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7299999999999999e-06 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495047+0000", 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4066000000000001e-05 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495089+0000", 2024-09-04T21:54:12.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2700000000000001e-05 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495112+0000", 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2965000000000001e-05 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495230+0000", 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117581 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495238+0000", 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.388e-06 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498788+0000", 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035494939999999998 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498872+0000", 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4190999999999994e-05 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498983+0000", 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110739 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.499844+0000", 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086077500000000004 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500358+0000", 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051428599999999997 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500414+0000", 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5689999999999997e-05 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500457+0000", 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3492999999999998e-05 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7675 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1179648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:12.524 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6843798290000001, 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057294420000000004, 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7675 2024-09-04T21:54:12.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735829+0000", 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8079999999999994e-06 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736259+0000", 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042982200000000001 2024-09-04T21:54:12.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736276+0000", 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7124000000000001e-05 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736287+0000", 2024-09-04T21:54:12.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0899e-05 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736380+0000", 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3304e-05 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736530+0000", 2024-09-04T21:54:12.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014963300000000001 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736871+0000", 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034072099999999999 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739225+0000", 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023541030000000002 2024-09-04T21:54:12.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741419+0000", 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021947189999999999 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741495+0000", 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5087999999999997e-05 2024-09-04T21:54:12.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741550+0000", 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5220999999999999e-05 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.532 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7676 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1164033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6814078759999997, 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00344502, 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.533 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7676 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738795+0000", 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3180000000000001e-06 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739162+0000", 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036728900000000003 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739179+0000", 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7079999999999999e-05 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739189+0000", 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9930000000000007e-06 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739281+0000", 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1866000000000004e-05 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739489+0000", 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020804800000000001 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739790+0000", 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030148600000000001 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741943+0000", 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021526610000000002 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742126+0000", 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018295599999999999 2024-09-04T21:54:12.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742191+0000", 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5234000000000006e-05 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742237+0000", 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6088999999999999e-05 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7824 3.a 3:572c7a4a:::rbd_header.1188e197bdd2:head [create,call rbd.create in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.910445465, 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030039490000000001, 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.540 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7824 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509758+0000", 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4800000000000001e-06 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509771+0000", 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3073e-05 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509806+0000", 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4913999999999998e-05 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509828+0000", 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2209e-05 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509960+0000", 2024-09-04T21:54:12.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131874 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510196+0000", 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023616500000000001 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510538+0000", 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034151599999999999 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.511889+0000", 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001351217 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512597+0000", 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070789799999999999 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512680+0000", 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3138000000000003e-05 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512759+0000", 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8465000000000002e-05 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7874 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8412515190000001, 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0051755769999999998, 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7874 2024-09-04T21:54:12.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578951+0000", 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9360000000000002e-06 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578958+0000", 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578977+0000", 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9011000000000001e-05 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578987+0000", 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3950000000000007e-06 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579158+0000", 2024-09-04T21:54:12.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000171266 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579330+0000", 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017195599999999999 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579662+0000", 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033173499999999999 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.583052+0000", 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033902189999999999 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584018+0000", 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096648399999999998 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584074+0000", 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5513999999999997e-05 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584124+0000", 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0441e-05 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7875 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8410589420000001, 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052294439999999998, 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7875 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.554 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579142+0000", 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0500000000000002e-07 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579147+0000", 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1120000000000004e-06 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579154+0000", 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.097e-06 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579159+0000", 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4870000000000004e-06 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581460+0000", 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023010230000000001 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581541+0000", 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0939999999999994e-05 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581739+0000", 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019828000000000001 2024-09-04T21:54:12.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584176+0000", 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002437051 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584259+0000", 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2831999999999996e-05 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:12.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584320+0000", 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1204000000000006e-05 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584371+0000", 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0612999999999998e-05 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4488.0:7874 3.a e32/22 ondisk, result = 0)", 2024-09-04T21:54:12.560 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8389189350000001, 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028556430000000002, 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581283+0000", 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5650000000000001e-06 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581289+0000", 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1940000000000003e-06 2024-09-04T21:54:12.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581304+0000", 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5386000000000001e-05 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581314+0000", 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2609999999999999e-06 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:12.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584004+0000", 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002690539 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584013+0000", 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3459999999999994e-06 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584137+0000", 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012435199999999999 2024-09-04T21:54:12.564 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.565 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.565 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.565 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:12.565 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:12.565 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:12.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:54:12.566 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:12.419+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:12.568 INFO:teuthology.orchestra.run.smithi112.stderr:marked out osd.1. 2024-09-04T21:54:12.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:12.569 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:12.563+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:12.667 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:12.667+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:12.741 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:12.741 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:12.742 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:12.742 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:12.758 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:12.768 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:12.766+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:12.869 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:12.867+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:12.919 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:12.919 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:12.933 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:12.970 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:12.970+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:13.071 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:13.070+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:13.102 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4259121400000003, 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4258853550000001, 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4258564219999998, 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4258327749999999, 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4258129940000002, 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4257935990000004, 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.425774326, 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4257551040000003, 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4257362669999996, 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.425716864, 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4256958080000004, 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4256636299999998, 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4256374090000001, 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4256092000000002, 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4255891600000004, 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4255697779999998, 2024-09-04T21:54:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4255461900000004, 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4255258729999998, 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4255063210000003, 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:13.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.195 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4254865619999997, 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:13.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.200 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:13.202 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:13.202 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:13.199+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:13.303 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:13.302+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:13.370 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:13.370 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:13.370 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:13.370 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:13.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:13.403 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:13.402+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:13.461 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:54:13.504 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:13.502+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:13.511 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:13.511 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:13.527 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:13.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:13.606 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:13.606+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:13.692 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:13.692 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.840936977, 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.693 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:13.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.714889714, 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:13.699 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.518336076000001, 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.490946578000001, 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.467682107, 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.439924316999999, 2024-09-04T21:54:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.439808755, 2024-09-04T21:54:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.13963397, 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.9805014180000002, 2024-09-04T21:54:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.9802753600000003, 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.5030622920000001, 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.2736741809999996, 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.9754835399999999, 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.9753706849999997, 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.9657097170000002, 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.9458599919999999, 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7956313059999998, 2024-09-04T21:54:13.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1293044009999997, 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1261652150000003, 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:13.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1078542249999996, 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:13.803 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:13.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:13.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:13.808 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:13.808 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:13.809 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:13.810 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:13.810 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:13.810 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:13.811 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:13.811 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:13.811 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:13.811 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:13.812 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:13.812 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:13.812 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:13.813 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:13.813 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:13.813 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:13.807+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:13.824 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:13.825 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:13.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:13.910 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:13.907+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:13.941 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:13.941 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:13.955 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:14.048 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:14.047+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:14.111 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:14.107+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:14.126 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:14.126 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:14.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:14.228 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:14.227+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:14.308 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:14.308 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:14.313 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:14.311+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:14.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:14.413 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:14.410+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:14.492 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:14.492 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:14.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:14.514 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:14.514+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:14.615 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:14.614+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:14.621 INFO:teuthology.orchestra.run.smithi112.stdout:flattening image image_client.0-clone5 2024-09-04T21:54:14.635 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:14.635 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:14.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:14.715 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:14.714+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:14.779 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:14.779 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:14.794 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:14.815 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:14.815+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:14.823 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-09-04T21:54:14.917 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:14.915+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:14.931 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:14.931 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:14.951 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:15.017 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:15.014+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:15.118 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:15.115+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:15.125 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:15.125 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:15.139 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:15.226 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:15.226+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:15.306 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:15.306 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:15.320 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:15.321 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:15.318+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:15.421 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:15.418+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:15.471 INFO:teuthology.orchestra.run.smithi112.stdout:1336 punch from 0x63715c2 to 0x637fe3f, (0xe87d bytes) 2024-09-04T21:54:15.484 INFO:teuthology.orchestra.run.smithi112.stdout:1337 read 0x58e111 thru 0x5928f6 (0x47e6 bytes) 2024-09-04T21:54:15.484 INFO:teuthology.orchestra.run.smithi112.stdout:1338 writesame 0x704e1a thru 0x71169c data_size 0x1(0xc883 bytes) 2024-09-04T21:54:15.487 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:15.487 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:15.495 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:54:15.502 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:15.521 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.522 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:15.519+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:15.529 INFO:teuthology.orchestra.run.smithi112.stdout:1340 trunc from 0xbe1e300 to 0x5c053b8 2024-09-04T21:54:15.558 INFO:teuthology.orchestra.run.smithi112.stdout:1341 read 0x11492d8 thru 0x114cc16 (0x393f bytes) 2024-09-04T21:54:15.559 INFO:teuthology.orchestra.run.smithi112.stdout:1342 read 0x5b0abd6 thru 0x5b0af58 (0x383 bytes) 2024-09-04T21:54:15.559 INFO:teuthology.orchestra.run.smithi112.stdout:1343 write 0x25f3fb7 thru 0x25f9a23 (0x5a6d bytes) 2024-09-04T21:54:15.564 INFO:teuthology.orchestra.run.smithi112.stdout:1344 trunc from 0x5c053b8 to 0x797f66b 2024-09-04T21:54:15.579 INFO:teuthology.orchestra.run.smithi112.stdout:1347 trunc from 0x797f66b to 0x27fcc5b 2024-09-04T21:54:15.589 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.605 INFO:teuthology.orchestra.run.smithi112.stdout:1349 write 0x9e9bbc5 thru 0x9ea2fbc (0x73f8 bytes) 2024-09-04T21:54:15.607 INFO:teuthology.orchestra.run.smithi112.stdout:1350 punch from 0x7745944 to 0x774fb11, (0xa1cd bytes) 2024-09-04T21:54:15.615 INFO:teuthology.orchestra.run.smithi112.stdout:1351 trunc from 0x9ea2fbd to 0x38aec26 2024-09-04T21:54:15.623 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:15.622+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:15.636 INFO:teuthology.orchestra.run.smithi112.stdout:1352 writesame 0x741965c thru 0x7424a77 data_size 0x2(0xb41c bytes) 2024-09-04T21:54:15.638 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.638 INFO:teuthology.orchestra.run.smithi112.stdout:1355 punch from 0x309c1ea to 0x30a8f1d, (0xcd33 bytes) 2024-09-04T21:54:15.650 INFO:teuthology.orchestra.run.smithi112.stdout:1356 writesame 0xbdccb46 thru 0xbdd0707 data_size 0x1(0x3bc2 bytes) 2024-09-04T21:54:15.663 INFO:teuthology.orchestra.run.smithi112.stdout:1358 write 0xd01254d thru 0xd0220e6 (0xfb9a bytes) 2024-09-04T21:54:15.672 INFO:teuthology.orchestra.run.smithi112.stdout:1359 trunc from 0xd0220e7 to 0xb518ab4 2024-09-04T21:54:15.681 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:15.682 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:15.683 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.684 INFO:teuthology.orchestra.run.smithi112.stdout:1361 trunc from 0xb518ab4 to 0xc50068a 2024-09-04T21:54:15.694 INFO:teuthology.orchestra.run.smithi112.stdout:1362 punch from 0x14ae560 to 0x14bc12d, (0xdbcd bytes) 2024-09-04T21:54:15.694 INFO:teuthology.orchestra.run.smithi112.stdout:1363 write 0x608ede3 thru 0x60960f3 (0x7311 bytes) 2024-09-04T21:54:15.697 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.697 INFO:teuthology.orchestra.run.smithi112.stdout:1366 read 0x52d3c06 thru 0x52d4298 (0x693 bytes) 2024-09-04T21:54:15.698 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:15.699 INFO:teuthology.orchestra.run.smithi112.stdout:1367 trunc from 0xc50068a to 0xda3f2cd 2024-09-04T21:54:15.707 INFO:teuthology.orchestra.run.smithi112.stdout:1368 writesame 0xbe83f15 thru 0xbe8bbd0 data_size 0x12(0x7cbc bytes) 2024-09-04T21:54:15.710 INFO:teuthology.orchestra.run.smithi112.stdout:1373 read 0x9596d1f thru 0x959bc29 (0x4f0b bytes) 2024-09-04T21:54:15.710 INFO:teuthology.orchestra.run.smithi112.stdout:1374 write 0x505e529 thru 0x506b4fb (0xcfd3 bytes) 2024-09-04T21:54:15.713 INFO:teuthology.orchestra.run.smithi112.stdout:1375 write 0x1f0cb9 thru 0x200ab4 (0xfdfc bytes) 2024-09-04T21:54:15.715 INFO:teuthology.orchestra.run.smithi112.stdout:1377 read 0xa176965 thru 0xa17e626 (0x7cc2 bytes) 2024-09-04T21:54:15.715 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.716 INFO:teuthology.orchestra.run.smithi112.stdout:1382 write 0x772b974 thru 0x772e271 (0x28fe bytes) 2024-09-04T21:54:15.717 INFO:teuthology.orchestra.run.smithi112.stdout:1383 write 0x51494bd thru 0x514b115 (0x1c59 bytes) 2024-09-04T21:54:15.718 INFO:teuthology.orchestra.run.smithi112.stdout:1384 read 0x4934bac thru 0x493f069 (0xa4be bytes) 2024-09-04T21:54:15.719 INFO:teuthology.orchestra.run.smithi112.stdout:1386 write 0x93c88ac thru 0x93d6b1d (0xe272 bytes) 2024-09-04T21:54:15.721 INFO:teuthology.orchestra.run.smithi112.stdout:1387 writesame 0x73f3b3e thru 0x73f6b8f data_size 0x5(0x3052 bytes) 2024-09-04T21:54:15.724 INFO:teuthology.orchestra.run.smithi112.stdout:1388 read 0xad44872 thru 0xad4b371 (0x6b00 bytes) 2024-09-04T21:54:15.724 INFO:teuthology.orchestra.run.smithi112.stdout:1389 writesame 0xa443f17 thru 0xa44b7b9 data_size 0x1(0x78a3 bytes) 2024-09-04T21:54:15.742 INFO:teuthology.orchestra.run.smithi112.stdout:1390 write 0x392a1de thru 0x392a450 (0x273 bytes) 2024-09-04T21:54:15.743 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:15.742+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:15.745 INFO:teuthology.orchestra.run.smithi112.stdout:1391 read 0x2b63343 thru 0x2b63724 (0x3e2 bytes) 2024-09-04T21:54:15.745 INFO:teuthology.orchestra.run.smithi112.stdout:1392 read 0xb7a5e6d thru 0xb7aa28b (0x441f bytes) 2024-09-04T21:54:15.745 INFO:teuthology.orchestra.run.smithi112.stdout:1393 read 0x90042ef thru 0x9008017 (0x3d29 bytes) 2024-09-04T21:54:15.746 INFO:teuthology.orchestra.run.smithi112.stdout:1394 write 0xac717f7 thru 0xac7e5f0 (0xcdfa bytes) 2024-09-04T21:54:15.750 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.750 INFO:teuthology.orchestra.run.smithi112.stdout:1397 write 0x676048d thru 0x676e48a (0xdffe bytes) 2024-09-04T21:54:15.753 INFO:teuthology.orchestra.run.smithi112.stdout:1398 read 0x28cb4d5 thru 0x28d057b (0x50a7 bytes) 2024-09-04T21:54:15.754 INFO:teuthology.orchestra.run.smithi112.stdout:1400 write 0x74b05af thru 0x74b2e8e (0x28e0 bytes) 2024-09-04T21:54:15.755 INFO:teuthology.orchestra.run.smithi112.stdout:1402 read 0xa9587c4 thru 0xa960c8a (0x84c7 bytes) 2024-09-04T21:54:15.756 INFO:teuthology.orchestra.run.smithi112.stdout:1404 write 0xe85737 thru 0xe9406f (0xe939 bytes) 2024-09-04T21:54:15.758 INFO:teuthology.orchestra.run.smithi112.stdout:1405 write 0x865a28 thru 0x86ee5e (0x9437 bytes) 2024-09-04T21:54:15.761 INFO:teuthology.orchestra.run.smithi112.stdout:1406 read 0xabad831 thru 0xabb5652 (0x7e22 bytes) 2024-09-04T21:54:15.761 INFO:teuthology.orchestra.run.smithi112.stdout:1407 read 0xb434f09 thru 0xb436fff (0x20f7 bytes) 2024-09-04T21:54:15.761 INFO:teuthology.orchestra.run.smithi112.stdout:1408 read 0xcc2e3ec thru 0xcc3bc42 (0xd857 bytes) 2024-09-04T21:54:15.761 INFO:teuthology.orchestra.run.smithi112.stdout:1409 read 0xcd072eb thru 0xcd09b02 (0x2818 bytes) 2024-09-04T21:54:15.771 INFO:teuthology.orchestra.run.smithi112.stdout:1410 writesame 0xed99387 thru 0xed9f67d data_size 0xf(0x62f7 bytes) 2024-09-04T21:54:15.771 INFO:teuthology.orchestra.run.smithi112.stdout:1412 writesame 0xe12fefd thru 0xe13d354 data_size 0x6(0xd458 bytes) 2024-09-04T21:54:15.777 INFO:teuthology.orchestra.run.smithi112.stdout:1413 writesame 0x1397234 thru 0x139ef45 data_size 0x7(0x7d12 bytes) 2024-09-04T21:54:15.781 INFO:teuthology.orchestra.run.smithi112.stdout:1414 punch from 0xa77076e to 0xa77f7c9, (0xf05b bytes) 2024-09-04T21:54:15.781 INFO:teuthology.orchestra.run.smithi112.stdout:1415 read 0xeaca99e thru 0xead08f2 (0x5f55 bytes) 2024-09-04T21:54:15.781 INFO:teuthology.orchestra.run.smithi112.stdout:1416 write 0x3544262 thru 0x3553db9 (0xfb58 bytes) 2024-09-04T21:54:15.783 INFO:teuthology.orchestra.run.smithi112.stdout:1418 write 0x9dd3942 thru 0x9de23f9 (0xeab8 bytes) 2024-09-04T21:54:15.786 INFO:teuthology.orchestra.run.smithi112.stdout:1419 read 0xcd24d8 thru 0xcd3f9e (0x1ac7 bytes) 2024-09-04T21:54:15.786 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.787 INFO:teuthology.orchestra.run.smithi112.stdout:1422 write 0xb5503b5 thru 0xb559818 (0x9464 bytes) 2024-09-04T21:54:15.790 INFO:teuthology.orchestra.run.smithi112.stdout:1423 write 0x9ee7088 thru 0x9eed4c0 (0x6439 bytes) 2024-09-04T21:54:15.792 INFO:teuthology.orchestra.run.smithi112.stdout:1424 read 0x7810202 thru 0x7811f86 (0x1d85 bytes) 2024-09-04T21:54:15.793 INFO:teuthology.orchestra.run.smithi112.stdout:1425 write 0x41bcf91 thru 0x41c6436 (0x94a6 bytes) 2024-09-04T21:54:15.796 INFO:teuthology.orchestra.run.smithi112.stdout:1426 read 0x14ad076 thru 0x14bb1e9 (0xe174 bytes) 2024-09-04T21:54:15.797 INFO:teuthology.orchestra.run.smithi112.stdout:1429 write 0xecae03c thru 0xecb6e13 (0x8dd8 bytes) 2024-09-04T21:54:15.801 INFO:teuthology.orchestra.run.smithi112.stdout:1430 read 0xc31aa66 thru 0xc31ea69 (0x4004 bytes) 2024-09-04T21:54:15.802 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.802 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.802 INFO:teuthology.orchestra.run.smithi112.stdout:1434 writesame 0xc3da69e thru 0xc3e329a data_size 0x1(0x8bfd bytes) 2024-09-04T21:54:15.825 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:15.823+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:15.832 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:15.835 INFO:teuthology.orchestra.run.smithi112.stdout:1435 trunc from 0xed9f67e to 0xbf64806 2024-09-04T21:54:15.846 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.846 INFO:teuthology.orchestra.run.smithi112.stdout:1437 read 0x87293e3 thru 0x872ef06 (0x5b24 bytes) 2024-09-04T21:54:15.859 INFO:teuthology.orchestra.run.smithi112.stdout:1438 write 0xe48f614 thru 0xe490a1d (0x140a bytes) 2024-09-04T21:54:15.866 INFO:teuthology.orchestra.run.smithi112.stdout:1439 write 0xf39b8e thru 0xf4969f (0xfb12 bytes) 2024-09-04T21:54:15.868 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:15.868 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:15.876 INFO:teuthology.orchestra.run.smithi112.stdout:1443 trunc from 0xe490a1e to 0x84bffee 2024-09-04T21:54:15.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:15.894 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:15.894 INFO:teuthology.orchestra.run.smithi112.stdout:1446 clone 7 order 25 su 131072 sc 10 2024-09-04T21:54:15.930 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:15.930+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:15.978 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:15.979 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:15.995 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:16.003 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:16.004 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:16.011 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:16.011 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:16.018 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:16.026 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:16.023+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:16.027 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:16.027 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:16.045 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:16.045 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:16.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:16.065 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:16.083 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:16.084 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:16.084 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:16.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:16.132 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:16.131+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:16.188 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:16.188 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:16.188 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:16.189 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:16.189 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:16.202 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:16.228 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:16.227+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:16.324 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:16.324 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:16.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:16.324 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1045 3.10 3:08ed8dc3:::rbd_data.11882e693cd0.000000000000002f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.409101135, 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054299810000000004, 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.325 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1045 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912477+0000", 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8369999999999999e-06 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912492+0000", 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.428e-05 2024-09-04T21:54:16.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912529+0000", 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7673000000000003e-05 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912547+0000", 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7689e-05 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912710+0000", 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162526 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912993+0000", 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028371199999999998 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913324+0000", 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033046099999999999 2024-09-04T21:54:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915482+0000", 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021584989999999999 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917665+0000", 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021827449999999998 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917778+0000", 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000113063 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917904+0000", 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012549600000000001 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1101 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.901245093, 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0067943780000000002, 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1101 2024-09-04T21:54:16.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420333+0000", 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6169999999999998e-06 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420347+0000", 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-05 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420381+0000", 2024-09-04T21:54:16.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4122999999999998e-05 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420398+0000", 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6286999999999998e-05 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420568+0000", 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017088199999999999 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420589+0000", 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0513000000000002e-05 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424142+0000", 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035527359999999999 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424352+0000", 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021066300000000001 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424715+0000", 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000362233 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426277+0000", 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015626069999999999 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426969+0000", 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069220800000000004 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427060+0000", 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1057000000000006e-05 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427124+0000", 2024-09-04T21:54:16.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3522000000000006e-05 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1102 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_stripe_unit_count in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.894216050000001, 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026178130000000001, 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.339 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1102 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427359+0000", 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427361+0000", 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.384e-06 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427371+0000", 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0086e-05 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427394+0000", 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2643e-05 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427406+0000", 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2247e-05 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427515+0000", 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010909899999999999 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427646+0000", 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013051299999999999 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.428016+0000", 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037050700000000001 2024-09-04T21:54:16.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429054+0000", 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010376230000000001 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429807+0000", 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075308500000000002 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429894+0000", 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7216e-05 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.429976+0000", 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2410000000000005e-05 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1116 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [watch watch cookie 139952113516272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.345 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.874137727000001, 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029237209999999998, 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1116 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447437+0000", 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447439+0000", 2024-09-04T21:54:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447449+0000", 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8239999999999995e-06 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447481+0000", 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1733000000000003e-05 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447497+0000", 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5653000000000001e-05 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447642+0000", 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014499599999999999 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.447778+0000", 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013675199999999999 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.448006+0000", 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022790500000000001 2024-09-04T21:54:16.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.449250+0000", 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001244172 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450163+0000", 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00091221100000000001 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450248+0000", 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5753000000000004e-05 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.450361+0000", 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011235600000000001 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.352 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1192 3.a 3:52245ef8:::rbd_data.1188e7c968d9.00000000000001c6:head [set-alloc-hint object_size 524288 write_size 524288,write 131042~30] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.737393386000001, 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027730150000000002, 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1192 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584181+0000", 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584184+0000", 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8629999999999999e-06 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584198+0000", 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3961999999999999e-05 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584231+0000", 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2660000000000002e-05 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584250+0000", 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.908e-05 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584339+0000", 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9456000000000005e-05 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584499+0000", 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000159431 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.584675+0000", 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017591899999999999 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586136+0000", 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014611050000000001 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586833+0000", 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069698700000000004 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586903+0000", 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0216000000000007e-05 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.586954+0000", 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1335999999999998e-05 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1267 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.592308781, 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026536139999999999, 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.359 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1267 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729266+0000", 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729268+0000", 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3800000000000001e-06 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729279+0000", 2024-09-04T21:54:16.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0749e-05 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729309+0000", 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0029999999999999e-05 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729323+0000", 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.411e-05 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729437+0000", 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114115 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729610+0000", 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000172821 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.729965+0000", 2024-09-04T21:54:16.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035526999999999998 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.730787+0000", 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082178999999999998 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731730+0000", 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094278199999999997 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731845+0000", 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114935 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.731920+0000", 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4632e-05 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.365 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1279 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.557846002, 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027977710000000001, 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1279 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763729+0000", 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763731+0000", 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.537e-06 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763743+0000", 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1426999999999999e-05 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763775+0000", 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1925999999999999e-05 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763789+0000", 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4756999999999999e-05 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.763900+0000", 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011079800000000001 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.764069+0000", 2024-09-04T21:54:16.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169307 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.764651+0000", 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058184199999999995 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766213+0000", 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015619130000000001 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766364+0000", 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000150502 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766453+0000", 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8902999999999999e-05 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.766526+0000", 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3858999999999999e-05 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1338 3.10 3:0a516185:::rbd_data.1188e7c968d9.000000000000010d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.46794019, 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050744040000000002, 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.372 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1338 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853634+0000", 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853638+0000", 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853649+0000", 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1161999999999999e-05 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853682+0000", 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3127000000000003e-05 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853697+0000", 2024-09-04T21:54:16.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5318000000000001e-05 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853801+0000", 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010342500000000001 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.853935+0000", 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000133771 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.854097+0000", 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016247399999999999 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.855226+0000", 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011287210000000001 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858515+0000", 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032894249999999999 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858634+0000", 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119226 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.858709+0000", 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4468999999999995e-05 2024-09-04T21:54:16.378 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1401 3.10 3:0966b599:::rbd_data.1188e7c968d9.0000000000000170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.358829954000001, 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026384160000000002, 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.379 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1401 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962745+0000", 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962748+0000", 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962757+0000", 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1600000000000004e-06 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962781+0000", 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4426e-05 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962793+0000", 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1955e-05 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.962928+0000", 2024-09-04T21:54:16.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135282 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963165+0000", 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023646100000000001 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.963584+0000", 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041887000000000001 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965107+0000", 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015236970000000001 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965129+0000", 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.139e-05 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965258+0000", 2024-09-04T21:54:16.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012891199999999999 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.965383+0000", 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012537900000000001 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1612 3.10 3:0aa9c1cb:::rbd_data.1188e7c968d9.00000000000001a4:head [set-alloc-hint object_size 524288 write_size 524288,write 426878~31874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.977687291000001, 2024-09-04T21:54:16.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003599587, 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1612 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343887+0000", 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343891+0000", 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.23e-06 2024-09-04T21:54:16.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.343991+0000", 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010068000000000001 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344028+0000", 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6579000000000003e-05 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344049+0000", 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1002999999999999e-05 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344136+0000", 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7100000000000003e-05 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344241+0000", 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010485200000000001 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.344342+0000", 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100721 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.345192+0000", 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085025300000000003 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347429+0000", 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022367620000000002 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347462+0000", 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2960999999999998e-05 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.347487+0000", 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5446e-05 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.391 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1615 3.10 3:0aa9c1cb:::rbd_data.1188e7c968d9.00000000000001a4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.964874583, 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029358750000000001, 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1615 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356700+0000", 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356702+0000", 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1830000000000001e-06 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356711+0000", 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5469999999999997e-06 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356769+0000", 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7896000000000003e-05 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356789+0000", 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0239999999999999e-05 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356866+0000", 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6822000000000004e-05 2024-09-04T21:54:16.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.356941+0000", 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5223999999999997e-05 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.357037+0000", 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6063000000000005e-05 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.357780+0000", 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074262700000000004 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359531+0000", 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001751072 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359588+0000", 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6884e-05 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:03.359636+0000", 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8316999999999998e-05 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.398 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7222 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [watch watch cookie 139952180803216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.5044058220000007, 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031156790000000001, 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7222 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.399 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817169+0000", 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817171+0000", 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0600000000000002e-06 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817180+0000", 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3300000000000005e-06 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817213+0000", 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2750999999999999e-05 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817229+0000", 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5498e-05 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817353+0000", 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012423600000000001 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817584+0000", 2024-09-04T21:54:16.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000230938 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.817997+0000", 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041338900000000001 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.819260+0000", 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012629690000000001 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820086+0000", 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082551100000000002 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820175+0000", 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104999999999995e-05 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:06.820285+0000", 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109892 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.404 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7465 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1039991~8585] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.9486434100000007, 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030096060000000002, 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.405 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7465 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372931+0000", 2024-09-04T21:54:16.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372934+0000", 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372947+0000", 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3232e-05 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372962+0000", 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5656000000000001e-05 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.372970+0000", 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7500000000000003e-06 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373031+0000", 2024-09-04T21:54:16.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1197000000000004e-05 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373143+0000", 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011143599999999999 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.373258+0000", 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114842 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.374309+0000", 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010515570000000001 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375856+0000", 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001547185 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375909+0000", 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2335e-05 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.375941+0000", 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2110000000000003e-05 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7519 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:16.411 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8265457339999998, 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054280980000000001, 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7519 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495033+0000", 2024-09-04T21:54:16.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7299999999999999e-06 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495047+0000", 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4066000000000001e-05 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495089+0000", 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2700000000000001e-05 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495112+0000", 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2965000000000001e-05 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495230+0000", 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117581 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495238+0000", 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.388e-06 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498788+0000", 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035494939999999998 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498872+0000", 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4190999999999994e-05 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:16.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498983+0000", 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110739 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.499844+0000", 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086077500000000004 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500358+0000", 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051428599999999997 2024-09-04T21:54:16.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500414+0000", 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5689999999999997e-05 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500457+0000", 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3492999999999998e-05 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7675 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1179648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.5857543199999995, 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057294420000000004, 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7675 2024-09-04T21:54:16.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735829+0000", 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8079999999999994e-06 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736259+0000", 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042982200000000001 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736276+0000", 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7124000000000001e-05 2024-09-04T21:54:16.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736287+0000", 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0899e-05 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736380+0000", 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3304e-05 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736530+0000", 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014963300000000001 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736871+0000", 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034072099999999999 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739225+0000", 2024-09-04T21:54:16.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023541030000000002 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741419+0000", 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021947189999999999 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741495+0000", 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5087999999999997e-05 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741550+0000", 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5220999999999999e-05 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7676 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1164033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.5827823670000001, 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00344502, 2024-09-04T21:54:16.425 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7676 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738792+0000", 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.738795+0000", 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3180000000000001e-06 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739162+0000", 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036728900000000003 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739179+0000", 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7079999999999999e-05 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739189+0000", 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9930000000000007e-06 2024-09-04T21:54:16.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739281+0000", 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1866000000000004e-05 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739489+0000", 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020804800000000001 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739790+0000", 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030148600000000001 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741943+0000", 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021526610000000002 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742126+0000", 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018295599999999999 2024-09-04T21:54:16.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742191+0000", 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5234000000000006e-05 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.742237+0000", 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6088999999999999e-05 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7824 3.a 3:572c7a4a:::rbd_header.1188e197bdd2:head [create,call rbd.create in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.8118199559999999, 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030039490000000001, 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.432 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7824 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509755+0000", 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509758+0000", 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4800000000000001e-06 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509771+0000", 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3073e-05 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509806+0000", 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4913999999999998e-05 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509828+0000", 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2209e-05 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.509960+0000", 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131874 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510196+0000", 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023616500000000001 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.510538+0000", 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034151599999999999 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.511889+0000", 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001351217 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512597+0000", 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070789799999999999 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512680+0000", 2024-09-04T21:54:16.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3138000000000003e-05 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.512759+0000", 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8465000000000002e-05 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7874 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:16.438 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.7426260100000004, 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0051755769999999998, 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7874 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578949+0000", 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578951+0000", 2024-09-04T21:54:16.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9360000000000002e-06 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578958+0000", 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578977+0000", 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9011000000000001e-05 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.578987+0000", 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3950000000000007e-06 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579158+0000", 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000171266 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579330+0000", 2024-09-04T21:54:16.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017195599999999999 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579662+0000", 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033173499999999999 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.583052+0000", 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033902189999999999 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584018+0000", 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096648399999999998 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584074+0000", 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5513999999999997e-05 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584124+0000", 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0441e-05 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.444 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7875 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.7424334330000004, 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052294439999999998, 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:16.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7875 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579142+0000", 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0500000000000002e-07 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579147+0000", 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1120000000000004e-06 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579154+0000", 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.097e-06 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579159+0000", 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4870000000000004e-06 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581460+0000", 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023010230000000001 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581541+0000", 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0939999999999994e-05 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581739+0000", 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019828000000000001 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584176+0000", 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002437051 2024-09-04T21:54:16.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584259+0000", 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2831999999999996e-05 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584320+0000", 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1204000000000006e-05 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584371+0000", 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0612999999999998e-05 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4488.0:7874 3.a e32/22 ondisk, result = 0)", 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.740293426, 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028556430000000002, 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:16.451 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581281+0000", 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581283+0000", 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5650000000000001e-06 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581289+0000", 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1940000000000003e-06 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581304+0000", 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5386000000000001e-05 2024-09-04T21:54:16.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581314+0000", 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2609999999999999e-06 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584004+0000", 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002690539 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:16.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584013+0000", 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3459999999999994e-06 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584137+0000", 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012435199999999999 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:16.455 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:54:16.458 INFO:teuthology.orchestra.run.smithi112.stdout:leaving image image_client.0-clone6 intact 2024-09-04T21:54:16.458 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:16.458 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:16.455+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:16.558 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:16.555+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:16.622 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-09-04T21:54:16.640 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:16.640 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:16.640 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:16.640 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:16.658 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:16.658+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:16.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:16.759 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:16.758+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:16.844 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:16.845 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:16.845 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:16.845 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:16.845 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:16.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:16.861 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:16.858+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:16.961 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:16.959+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:16.976 INFO:teuthology.orchestra.run.smithi112.stdout:1447 write 0x63e0f17 thru 0x63e67c9 (0x58b3 bytes) 2024-09-04T21:54:16.988 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:16.988 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312380469000001, 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:16.989 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312353684, 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312324751, 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312301103999999, 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.004 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:17.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:17.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312281323000001, 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312261928, 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312242655, 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312223433, 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312204596000001, 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:17.028 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:17.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312185192999999, 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.033 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:17.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.037 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312164137, 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:17.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312131959, 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.042 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:17.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:17.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312105738, 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:17.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:17.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.051 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312077529, 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:17.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312057489000001, 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.056 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:17.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.060 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312038106999999, 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.061 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:17.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:17.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.312014519, 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:17.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:17.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:17.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.311994201999999, 2024-09-04T21:54:17.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:17.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:17.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.31197465, 2024-09-04T21:54:17.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:17.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:17.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.311954890999999, 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.085 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:17.086 INFO:teuthology.orchestra.run.smithi112.stdout:1448 write 0x750bd33 thru 0x7515482 (0x9750 bytes) 2024-09-04T21:54:17.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:17.088 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.088 INFO:teuthology.orchestra.run.smithi112.stdout:1453 read 0x447a288 thru 0x447d1a6 (0x2f1f bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1454 write 0x6ecc466 thru 0x6ed7bff (0xb79a bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1455 punch from 0x5847c82 to 0x5857676, (0xf9f4 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1456 read 0x64f2242 thru 0x64fc402 (0xa1c1 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1457 read 0x48ad748 thru 0x48b995a (0xc213 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1458 write 0xd747943 thru 0xd755fa7 (0xe665 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1459 writesame 0xd2bcaa6 thru 0xd2bd92c data_size 0x1(0xe87 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1461 read 0x1d663a7 thru 0x1d70a6f (0xa6c9 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1462 read 0x653a57 thru 0x6602b6 (0xc860 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1463 writesame 0x94a1cff thru 0x94a8edf data_size 0x1(0x71e1 bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1464 write 0x6893bbe thru 0x689de0a (0xa24d bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1465 read 0xc643e8e thru 0xc64fb7a (0xbced bytes) 2024-09-04T21:54:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:1466 punch from 0x4e7be79 to 0x4e7da32, (0x1bb9 bytes) 2024-09-04T21:54:17.090 INFO:teuthology.orchestra.run.smithi112.stdout:1467 punch from 0x8dce17a to 0x8dd69e6, (0x886c bytes) 2024-09-04T21:54:17.090 INFO:teuthology.orchestra.run.smithi112.stdout:1469 read 0x1da3d84 thru 0x1db3826 (0xfaa3 bytes) 2024-09-04T21:54:17.090 INFO:teuthology.orchestra.run.smithi112.stdout:1472 writesame 0x65bf972 thru 0x65c9636 data_size 0x1(0x9cc5 bytes) 2024-09-04T21:54:17.090 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:17.086+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:17.127 INFO:teuthology.orchestra.run.smithi112.stdout:1474 punch from 0xb2cba93 to 0xb2cdb00, (0x206d bytes) 2024-09-04T21:54:17.127 INFO:teuthology.orchestra.run.smithi112.stdout:1475 write 0x734d935 thru 0x7353d25 (0x63f1 bytes) 2024-09-04T21:54:17.131 INFO:teuthology.orchestra.run.smithi112.stdout:1476 write 0x9e186c thru 0x9e6fa6 (0x573b bytes) 2024-09-04T21:54:17.157 INFO:teuthology.orchestra.run.smithi112.stdout:1477 trunc from 0xd755fa8 to 0x4cbf31c 2024-09-04T21:54:17.187 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:17.186+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:17.193 INFO:teuthology.orchestra.run.smithi112.stdout:1478 trunc from 0x4cbf31c to 0xb2e0553 2024-09-04T21:54:17.201 INFO:teuthology.orchestra.run.smithi112.stdout:1479 write 0x60371ac thru 0x6040c8e (0x9ae3 bytes) 2024-09-04T21:54:17.204 INFO:teuthology.orchestra.run.smithi112.stdout:1480 read 0x3e7f734 thru 0x3e852e1 (0x5bae bytes) 2024-09-04T21:54:17.205 INFO:teuthology.orchestra.run.smithi112.stdout:1481 punch from 0x9920e4 to 0x9997f1, (0x770d bytes) 2024-09-04T21:54:17.213 INFO:teuthology.orchestra.run.smithi112.stdout:1482 write 0xbf9e3f0 thru 0xbfac688 (0xe299 bytes) 2024-09-04T21:54:17.216 INFO:teuthology.orchestra.run.smithi112.stdout:1484 punch from 0x5adbd2b to 0x5ae5343, (0x9618 bytes) 2024-09-04T21:54:17.235 INFO:teuthology.orchestra.run.smithi112.stdout:1485 trunc from 0xbfac689 to 0x3e0abd6 2024-09-04T21:54:17.251 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.251 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.272 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:17.272 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:17.274 INFO:teuthology.orchestra.run.smithi112.stdout:1490 writesame 0xbc59258 thru 0xbc6794c data_size 0x19(0xe6f5 bytes) 2024-09-04T21:54:17.277 INFO:teuthology.orchestra.run.smithi112.stdout:1492 writesame 0x9f1c41e thru 0x9f1fb54 data_size 0xb(0x3737 bytes) 2024-09-04T21:54:17.281 INFO:teuthology.orchestra.run.smithi112.stdout:1494 read 0x700a68f thru 0x700e593 (0x3f05 bytes) 2024-09-04T21:54:17.282 INFO:teuthology.orchestra.run.smithi112.stdout:1496 read 0x311a732 thru 0x3127c89 (0xd558 bytes) 2024-09-04T21:54:17.283 INFO:teuthology.orchestra.run.smithi112.stdout:1497 write 0x43feb04 thru 0x4404dde (0x62db bytes) 2024-09-04T21:54:17.286 INFO:teuthology.orchestra.run.smithi112.stdout:1498 read 0xa36b5d9 thru 0xa36b9d7 (0x3ff bytes) 2024-09-04T21:54:17.288 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:17.287+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:17.289 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:17.297 INFO:teuthology.orchestra.run.smithi112.stdout:1499 trunc from 0xbc6794d to 0x7856478 2024-09-04T21:54:17.334 INFO:teuthology.orchestra.run.smithi112.stdout:1501 write 0xe59a74f thru 0xe5a89bc (0xe26e bytes) 2024-09-04T21:54:17.338 INFO:teuthology.orchestra.run.smithi112.stdout:1504 writesame 0x87c07f1 thru 0x87c4f09 data_size 0x1(0x4719 bytes) 2024-09-04T21:54:17.348 INFO:teuthology.orchestra.run.smithi112.stdout:1506 read 0x3b64511 thru 0x3b6c7a5 (0x8295 bytes) 2024-09-04T21:54:17.349 INFO:teuthology.orchestra.run.smithi112.stdout:1507 write 0x4231dcf thru 0x42376da (0x590c bytes) 2024-09-04T21:54:17.352 INFO:teuthology.orchestra.run.smithi112.stdout:1508 writesame 0x373d998 thru 0x3749c7e data_size 0x11(0xc2e7 bytes) 2024-09-04T21:54:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:1511 punch from 0x5536ffe to 0x553d9c3, (0x69c5 bytes) 2024-09-04T21:54:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:1514 write 0x1dd0bed thru 0x1dd570e (0x4b22 bytes) 2024-09-04T21:54:17.360 INFO:teuthology.orchestra.run.smithi112.stdout:1515 punch from 0x5c5e825 to 0x5c6bcdb, (0xd4b6 bytes) 2024-09-04T21:54:17.361 INFO:teuthology.orchestra.run.smithi112.stdout:1516 write 0xc11db11 thru 0xc128c57 (0xb147 bytes) 2024-09-04T21:54:17.364 INFO:teuthology.orchestra.run.smithi112.stdout:1518 write 0xe274d01 thru 0xe281655 (0xc955 bytes) 2024-09-04T21:54:17.367 INFO:teuthology.orchestra.run.smithi112.stdout:1519 read 0x41cd7ea thru 0x41da2eb (0xcb02 bytes) 2024-09-04T21:54:17.368 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.368 INFO:teuthology.orchestra.run.smithi112.stdout:1521 write 0x2b79f9b thru 0x2b895ad (0xf613 bytes) 2024-09-04T21:54:17.382 INFO:teuthology.orchestra.run.smithi112.stdout:1523 trunc from 0xe5a89bd to 0x7d41d16 2024-09-04T21:54:17.389 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:17.387+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:17.410 INFO:teuthology.orchestra.run.smithi112.stdout:1525 write 0xc10c561 thru 0xc10f3ec (0x2e8c bytes) 2024-09-04T21:54:17.413 INFO:teuthology.orchestra.run.smithi112.stdout:1527 read 0x436e9fe thru 0x43749f6 (0x5ff9 bytes) 2024-09-04T21:54:17.414 INFO:teuthology.orchestra.run.smithi112.stdout:1528 punch from 0x5c49077 to 0x5c5550c, (0xc495 bytes) 2024-09-04T21:54:17.414 INFO:teuthology.orchestra.run.smithi112.stdout:1529 write 0x5ebc1f thru 0x5eeb34 (0x2f16 bytes) 2024-09-04T21:54:17.426 INFO:teuthology.orchestra.run.smithi112.stdout:1530 writesame 0xc7a51c1 thru 0xc7b0008 data_size 0x8f(0xae48 bytes) 2024-09-04T21:54:17.429 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.429 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.445 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:17.445 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:17.445 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:17.445 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:17.445 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:17.451 INFO:teuthology.orchestra.run.smithi112.stdout:1533 trunc from 0xc7b0009 to 0x17b20f7 2024-09-04T21:54:17.464 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:17.465 INFO:teuthology.orchestra.run.smithi112.stdout:1534 writesame 0x15261d9 thru 0x1534d24 data_size 0x4a(0xeb4c bytes) 2024-09-04T21:54:17.468 INFO:teuthology.orchestra.run.smithi112.stdout:1536 write 0xe2f0b4 thru 0xe32c77 (0x3bc4 bytes) 2024-09-04T21:54:17.480 INFO:teuthology.orchestra.run.smithi112.stdout:1537 writesame 0x224d8f5 thru 0x2253c3d data_size 0x1(0x6349 bytes) 2024-09-04T21:54:17.484 INFO:teuthology.orchestra.run.smithi112.stdout:1538 trunc from 0x2253c3e to 0x358bd05 2024-09-04T21:54:17.490 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:17.486+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:17.503 INFO:teuthology.orchestra.run.smithi112.stdout:1539 trunc from 0x358bd05 to 0x831647b 2024-09-04T21:54:17.511 INFO:teuthology.orchestra.run.smithi112.stdout:1540 writesame 0x82f5721 thru 0x8302d5e data_size 0xb(0xd63e bytes) 2024-09-04T21:54:17.516 INFO:teuthology.orchestra.run.smithi112.stdout:1541 writesame 0x214f879 thru 0x215de69 data_size 0x3d(0xe5f1 bytes) 2024-09-04T21:54:17.519 INFO:teuthology.orchestra.run.smithi112.stdout:1543 read 0x7c51cd3 thru 0x7c61a11 (0xfd3f bytes) 2024-09-04T21:54:17.520 INFO:teuthology.orchestra.run.smithi112.stdout:1545 write 0x28bb522 thru 0x28bf8df (0x43be bytes) 2024-09-04T21:54:17.523 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:17.524 INFO:teuthology.orchestra.run.smithi112.stdout:1547 read 0x41a8987 thru 0x41b3855 (0xaecf bytes) 2024-09-04T21:54:17.525 INFO:teuthology.orchestra.run.smithi112.stdout:1548 read 0x3638a5e thru 0x3643aee (0xb091 bytes) 2024-09-04T21:54:17.536 INFO:teuthology.orchestra.run.smithi112.stdout:1549 write 0xa370207 thru 0xa376660 (0x645a bytes) 2024-09-04T21:54:17.539 INFO:teuthology.orchestra.run.smithi112.stdout:1551 clone 8 order 24 su 262144 sc 6 2024-09-04T21:54:17.569 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-04T21:54:17.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:54:17.570 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-04T21:54:17.570 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-04T21:54:17.570 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-04T21:54:17.570 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-04T21:54:17.590 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:17.590 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:17.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:17.590 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:17.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.590 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.737662179000001, 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:17.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:17.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:17.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:17.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.611614916000001, 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.598 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:17.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:17.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.415061278, 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:17.604 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:17.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:17.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.38767178, 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:17.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:17.615 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.364407309000001, 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.616 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:17.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:17.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:17.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.622 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.336649519, 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.623 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:17.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:17.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.336533957, 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.629 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:17.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:17.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:17.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.036359171999999, 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.636 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:17.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:17.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.642 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.87722662, 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.643 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:17.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:17.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.877000561999999, 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.649 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:17.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:17.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.399787494, 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:17.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:17.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:17.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:17.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.662 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.170399382999999, 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:17.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8722087419999998, 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:17.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:17.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:17.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:17.676 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8720958870000004, 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:17.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:17.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:17.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.862434919, 2024-09-04T21:54:17.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:17.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:17.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:17.688 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8425851939999998, 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:17.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.6923565079999996, 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.0260296029999996, 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:17.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.0228904169999993, 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.706 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.0045794269999995, 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:17.715 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:17.716 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:17.587+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:17.717 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:17.718 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-04T21:54:17.718 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:17.719 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:17.718+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:17.819 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:17.818+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:17.887 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:17.887 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:17.887 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:17.887 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:17.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:17.921 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:17.918+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:18.018 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:18.019 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:18.019 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:18.019 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:18.019 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:18.022 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:18.019+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:18.032 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:18.089 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:18.123 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:18.122+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:18.204 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1291 3.4 e28/22 3:253162de:::rbd_data.1188e7c968d9.000000000000002f:head v 28'10, mlcod=27'8)", 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.784841+0000", 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.418642889999999, 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019663559999999998, 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.784841+0000", 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.784841+0000", 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.784842+0000", 2024-09-04T21:54:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.813e-06 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.784867+0000", 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4443999999999998e-05 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.784896+0000", 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8931000000000001e-05 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.784908+0000", 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1874e-05 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.785041+0000", 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000133912 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.785086+0000", 2024-09-04T21:54:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4044e-05 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.786730+0000", 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001644829 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.786807+0000", 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6508999999999995e-05 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1421 3.0 e28/22 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head v 28'14, mlcod=27'12)", 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.979684+0000", 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.223799097000001, 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025395589999999998, 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979684+0000", 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979684+0000", 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979686+0000", 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0660000000000002e-06 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980465+0000", 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00077882400000000001 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980505+0000", 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9603e-05 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980524+0000", 2024-09-04T21:54:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8736000000000002e-05 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980678+0000", 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015410200000000001 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980701+0000", 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3754999999999999e-05 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982122+0000", 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001420457 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982224+0000", 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010201599999999999 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1443 3.4 e28/22 3:23d7f9c7:::rbd_data.1188e7c968d9.0000000000000180:head v 28'13, mlcod=28'11)", 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.030308+0000", 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.173175430000001, 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002005953, 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030308+0000", 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030308+0000", 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030311+0000", 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.163e-06 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030351+0000", 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0160000000000002e-05 2024-09-04T21:54:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030395+0000", 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3732000000000001e-05 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030421+0000", 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5884000000000002e-05 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030549+0000", 2024-09-04T21:54:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012839200000000001 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.030606+0000", 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000003e-05 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.032205+0000", 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015989089999999999 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.032314+0000", 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108713 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1516 3.4 e28/22 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head v 28'16, mlcod=28'14)", 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.175411+0000", 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.028072377999999, 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021314749999999999, 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175411+0000", 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175411+0000", 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175414+0000", 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175462+0000", 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8439999999999997e-05 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175498+0000", 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5457000000000003e-05 2024-09-04T21:54:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175514+0000", 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6059e-05 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175670+0000", 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015652900000000001 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175737+0000", 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6166000000000003e-05 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177441+0000", 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001704002 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177543+0000", 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010189200000000001 2024-09-04T21:54:18.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1555 3.4 e28/22 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head v 28'17, mlcod=28'15)", 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.225928+0000", 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.97755564, 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002570541, 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.224 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225928+0000", 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225928+0000", 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225931+0000", 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7669999999999999e-06 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225963+0000", 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2095999999999999e-05 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226000+0000", 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7299000000000003e-05 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226022+0000", 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2459999999999998e-05 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226211+0000", 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000188655 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226279+0000", 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8003000000000006e-05 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228381+0000", 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021015919999999998 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228498+0000", 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117669 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1560 3.4 e28/22 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head v 28'18, mlcod=28'16)", 2024-09-04T21:54:18.228 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.253857+0000", 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.949626630999999, 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025741710000000001, 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253857+0000", 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253857+0000", 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253860+0000", 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9730000000000002e-06 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253877+0000", 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7353000000000001e-05 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253921+0000", 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3878999999999998e-05 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253962+0000", 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0763999999999998e-05 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254124+0000", 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000162282 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254202+0000", 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8262000000000004e-05 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256280+0000", 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020774119999999998 2024-09-04T21:54:18.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256431+0000", 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015124600000000001 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7156 3.4 e30/22 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head v 30'66, mlcod=30'64)", 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.490437904, 2024-09-04T21:54:18.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002957763, 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713047+0000", 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.528e-06 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713248+0000", 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201287 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713277+0000", 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8487e-05 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713288+0000", 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0832000000000001e-05 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713359+0000", 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0993000000000003e-05 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713382+0000", 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3580000000000001e-05 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715959+0000", 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025766869999999998 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716003+0000", 2024-09-04T21:54:18.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4369000000000002e-05 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7159 3.9 e30/22 3:93c2ae1d:::rbd_data.118867f9efa3.000000000000000f:head v 30'14, mlcod=29'10)", 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:18.238 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.488268290000001, 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031393580000000001, 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:18.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715216+0000", 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6300000000000004e-07 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715223+0000", 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2980000000000003e-06 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715237+0000", 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3361e-05 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715244+0000", 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.362e-06 2024-09-04T21:54:18.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716436+0000", 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00119187 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716456+0000", 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0562999999999999e-05 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718316+0000", 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018595720000000001 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718354+0000", 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8469e-05 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.243 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7367 3.9 e30/22 3:91a08b27:::rbd_data.1188b1294ada.0000000000000031:head v 30'15, mlcod=30'12)", 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.113334+0000", 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.090149208, 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022669959999999999, 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113334+0000", 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113334+0000", 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113337+0000", 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4260000000000002e-06 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113396+0000", 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8906999999999997e-05 2024-09-04T21:54:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113427+0000", 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1662999999999998e-05 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113442+0000", 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4805e-05 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113555+0000", 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112711 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.113593+0000", 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8603000000000003e-05 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.115464+0000", 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001870703 2024-09-04T21:54:18.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.115601+0000", 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013717800000000001 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7387 3.4 e30/22 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head v 30'72, mlcod=30'70)", 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.190538+0000", 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.012945116999999, 2024-09-04T21:54:18.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024016380000000002, 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190538+0000", 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190538+0000", 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190541+0000", 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0790000000000001e-06 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190577+0000", 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5246000000000001e-05 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190611+0000", 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4103000000000002e-05 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190626+0000", 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5495999999999999e-05 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190781+0000", 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015492199999999999 2024-09-04T21:54:18.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190850+0000", 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9048999999999996e-05 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.192845+0000", 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019943560000000001 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.192940+0000", 2024-09-04T21:54:18.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5387e-05 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7702 3.9 e31/22 3:9478b05c:::rbd_data.1188b1294ada.000000000000005b:head v 31'17, mlcod=30'15)", 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.809469+0000", 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.394014599, 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001956433, 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809469+0000", 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809469+0000", 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809469+0000", 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5700000000000002e-07 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809589+0000", 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000119889 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809612+0000", 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3031999999999999e-05 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809627+0000", 2024-09-04T21:54:18.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4294000000000001e-05 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809754+0000", 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127016 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.809792+0000", 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8086000000000001e-05 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.811306+0000", 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001514423 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.811425+0000", 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011913600000000001 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7729 3.4 e31/22 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head v 31'74, mlcod=30'72)", 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:18.257 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.30492317, 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031016809999999998, 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898561+0000", 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898620+0000", 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8813000000000001e-05 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898654+0000", 2024-09-04T21:54:18.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4292999999999998e-05 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898675+0000", 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1024000000000001e-05 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898834+0000", 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015876599999999999 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898899+0000", 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5209000000000005e-05 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901577+0000", 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026777849999999998 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901662+0000", 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4983999999999998e-05 2024-09-04T21:54:18.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7863 3.9 e32/22 3:9478b05c:::rbd_data.1188b1294ada.000000000000005b:head v 32'21, mlcod=31'18)", 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.6388524800000006, 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002738262, 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564632+0000", 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4000000000000001e-07 2024-09-04T21:54:18.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564643+0000", 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0966e-05 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564691+0000", 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8231e-05 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564714+0000", 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2857000000000002e-05 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.565011+0000", 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029731499999999998 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.565062+0000", 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1258000000000003e-05 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567253+0000", 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021909210000000002 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567369+0000", 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000115974 2024-09-04T21:54:18.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7862 3.4 e32/22 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head v 32'76, mlcod=30'73)", 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.6373849180000004, 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031099529999999999, 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566099+0000", 2024-09-04T21:54:18.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9999999999999996e-07 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566110+0000", 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1009000000000001e-05 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566141+0000", 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0244e-05 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566161+0000", 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0224000000000002e-05 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566407+0000", 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024641000000000002 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566463+0000", 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6113999999999998e-05 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569116+0000", 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026528519999999998 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569208+0000", 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2299999999999994e-05 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "MOSDPGPush(3.1a 36/35 [PushOp(3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:4, version: 29'20, data_included: [0~294912,305638~15526,327680~196608], data_size: 507046, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:4@29'20, size: 524288, copy_subset: [(0, 524288)], clone_subset: {}, snapset: 4=[]:{4: [4]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 524288, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:13.518723+0000", 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.6847605679999997, 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002408515, 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.518723+0000", 2024-09-04T21:54:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.518723+0000", 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.518724+0000", 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.043e-06 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.519700+0000", 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097581499999999999 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.519725+0000", 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5593e-05 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.519740+0000", 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4893e-05 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.519843+0000", 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102369 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.519855+0000", 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2224e-05 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:13.521131+0000", 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001276578 2024-09-04T21:54:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14326 3.0 3:0619fb3a:::rbd_header.118816801d55:head [watch watch cookie 139952114223744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.581142915, 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00266585, 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:18.276 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14326 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622342+0000", 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622350+0000", 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0190000000000006e-06 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622373+0000", 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2634999999999999e-05 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622385+0000", 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622460+0000", 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4231000000000001e-05 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622524+0000", 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4548000000000006e-05 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622642+0000", 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117707 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:18.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.623541+0000", 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00089878600000000005 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.624756+0000", 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012156300000000001 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.624853+0000", 2024-09-04T21:54:18.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6292000000000006e-05 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.625006+0000", 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015378000000000001 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14565 3.0 3:05912ce2:::rbd_data.118816801d55.00000000000000ee:head [sparse-read 0~524288] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:18.282 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.4817565450000001, 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030052669999999998, 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14565 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721728+0000", 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0669999999999999e-06 2024-09-04T21:54:18.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721733+0000", 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0189999999999997e-06 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721749+0000", 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6303e-05 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721757+0000", 2024-09-04T21:54:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7680000000000008e-06 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723062+0000", 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001305283 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.724732+0000", 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001669827 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14581 3.0 3:0280f30f:::rbd_data.118816801d55.00000000000000fe:head [sparse-read 0~524288] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.723011+0000", 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.4804719369999999, 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019830350000000002, 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14581 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723011+0000", 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723011+0000", 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723013+0000", 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3e-06 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723025+0000", 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2483000000000001e-05 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723052+0000", 2024-09-04T21:54:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6537999999999999e-05 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723066+0000", 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3919e-05 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.724770+0000", 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017039850000000001 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.724995+0000", 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022481000000000001 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14591 3.0 3:048868a5:::rbd_data.118816801d55.0000000000000108:head [sparse-read 0~524288] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.723220+0000", 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.480263082, 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024428470000000002, 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14591 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723220+0000", 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723220+0000", 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723221+0000", 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.065e-06 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723227+0000", 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.288e-06 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723240+0000", 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3743999999999999e-05 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723250+0000", 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3659999999999998e-06 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.725068+0000", 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018178090000000001 2024-09-04T21:54:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.725663+0000", 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059557499999999999 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14711 3.0 3:0619fb3a:::rbd_header.118816801d55:head [watch unwatch cookie 139952114223744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.3844766960000001, 2024-09-04T21:54:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028282910000000001, 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14711 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819010+0000", 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8569999999999999e-06 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819020+0000", 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0808999999999999e-05 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819049+0000", 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8697999999999999e-05 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819061+0000", 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1617000000000001e-05 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819171+0000", 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011018000000000001 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819302+0000", 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000130751 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819493+0000", 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019150100000000001 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.820539+0000", 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010458620000000001 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821701+0000", 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011623009999999999 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821758+0000", 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6988999999999999e-05 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821835+0000", 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6725999999999997e-05 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:18.301 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:18.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:18.303 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:18.304 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:18.304 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:18.304 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:18.305 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:18.305 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:18.305 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:18.299+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:18.315 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:18.335 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:18.336 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:18.356 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:18.356 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:18.377 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:18.378 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:18.378 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:18.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:18.424 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:18.423+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:18.432 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:18.432 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:18.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:18.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:54:18.505 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:18.502+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:18.606 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:18.602+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:18.615 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone7 from 0xa376661 (overlap 0x17b20f7) to 0x97a6c9 2024-09-04T21:54:18.620 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:18.620 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:18.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:18.707 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:18.703+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-09-04T21:54:18.759 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:18.760 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:18.777 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:18.820 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:18.819+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:18.908 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:18.908 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:18.909 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:18.907+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:18.922 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:18.967 INFO:teuthology.orchestra.run.smithi112.stdout:1555 trunc from 0xa376661 to 0x47c1683 2024-09-04T21:54:19.010 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:19.010+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:19.041 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:19.041 INFO:teuthology.orchestra.run.smithi112.stdout:1557 read 0x3655027 thru 0x3655f44 (0xf1e bytes) 2024-09-04T21:54:19.042 INFO:teuthology.orchestra.run.smithi112.stdout:1558 read 0x33a0a85 thru 0x33a7c9a (0x7216 bytes) 2024-09-04T21:54:19.053 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:19.053 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:19.062 INFO:teuthology.orchestra.run.smithi112.stdout:1559 writesame 0xbdfdcad thru 0xbe052dd data_size 0x1(0x7631 bytes) 2024-09-04T21:54:19.062 INFO:teuthology.orchestra.run.smithi112.stdout:1560 read 0x8c4a09f thru 0x8c4ff02 (0x5e64 bytes) 2024-09-04T21:54:19.062 INFO:teuthology.orchestra.run.smithi112.stdout:1561 write 0x723214 thru 0x727eb4 (0x4ca1 bytes) 2024-09-04T21:54:19.064 INFO:teuthology.orchestra.run.smithi112.stdout:1562 write 0x134ce6a thru 0x1355476 (0x860d bytes) 2024-09-04T21:54:19.067 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:19.067 INFO:teuthology.orchestra.run.smithi112.stdout:1564 read 0x2efac28 thru 0x2efd1f1 (0x25ca bytes) 2024-09-04T21:54:19.068 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:19.068 INFO:teuthology.orchestra.run.smithi112.stdout:1566 punch from 0x21a1895 to 0x21a1ca0, (0x40b bytes) 2024-09-04T21:54:19.068 INFO:teuthology.orchestra.run.smithi112.stdout:1567 punch from 0x44a79da to 0x44b631e, (0xe944 bytes) 2024-09-04T21:54:19.068 INFO:teuthology.orchestra.run.smithi112.stdout:1571 trunc from 0xbe052de to 0xc2b0c6b 2024-09-04T21:54:19.069 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:19.077 INFO:teuthology.orchestra.run.smithi112.stdout:1572 read 0xb81b5d thru 0xb8934b (0x77ef bytes) 2024-09-04T21:54:19.078 INFO:teuthology.orchestra.run.smithi112.stdout:1576 read 0xb528068 thru 0xb5284bd (0x456 bytes) 2024-09-04T21:54:19.079 INFO:teuthology.orchestra.run.smithi112.stdout:1577 write 0x9371445 thru 0x937e6a3 (0xd25f bytes) 2024-09-04T21:54:19.085 INFO:teuthology.orchestra.run.smithi112.stdout:1578 writesame 0x84b6ed8 thru 0x84bb777 data_size 0x53(0x48a0 bytes) 2024-09-04T21:54:19.091 INFO:teuthology.orchestra.run.smithi112.stdout:1580 read 0x909cb1a thru 0x90a8de8 (0xc2cf bytes) 2024-09-04T21:54:19.092 INFO:teuthology.orchestra.run.smithi112.stdout:1582 read 0x7c53142 thru 0x7c54d3e (0x1bfd bytes) 2024-09-04T21:54:19.107 INFO:teuthology.orchestra.run.smithi112.stdout:1583 write 0xe6007d6 thru 0xe60dafa (0xd325 bytes) 2024-09-04T21:54:19.110 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:19.110+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:19.112 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:19.113 INFO:teuthology.orchestra.run.smithi112.stdout:1586 read 0xbac6256 thru 0xbac8965 (0x2710 bytes) 2024-09-04T21:54:19.119 INFO:teuthology.orchestra.run.smithi112.stdout:1587 trunc from 0xe60dafb to 0xb964f41 2024-09-04T21:54:19.140 INFO:teuthology.orchestra.run.smithi112.stdout:1588 punch from 0x351a8bc to 0x351cd30, (0x2474 bytes) 2024-09-04T21:54:19.158 INFO:teuthology.orchestra.run.smithi112.stdout:1590 trunc from 0xb964f41 to 0x248e089 2024-09-04T21:54:19.212 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:19.210+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:19.214 INFO:teuthology.orchestra.run.smithi112.stdout:1591 write 0xd114190 thru 0xd122375 (0xe1e6 bytes) 2024-09-04T21:54:19.221 INFO:teuthology.orchestra.run.smithi112.stdout:1593 punch from 0xb6b4ad2 to 0xb6c027a, (0xb7a8 bytes) 2024-09-04T21:54:19.221 INFO:teuthology.orchestra.run.smithi112.stdout:1594 writesame 0x86c7211 thru 0x86d1678 data_size 0x4(0xa468 bytes) 2024-09-04T21:54:19.228 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:19.228 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:19.237 INFO:teuthology.orchestra.run.smithi112.stdout:1596 read 0x401209c thru 0x401989f (0x7804 bytes) 2024-09-04T21:54:19.238 INFO:teuthology.orchestra.run.smithi112.stdout:1597 write 0x47f2547 thru 0x47f7549 (0x5003 bytes) 2024-09-04T21:54:19.243 INFO:teuthology.orchestra.run.smithi112.stdout:1598 punch from 0x13691c3 to 0x136fbb6, (0x69f3 bytes) 2024-09-04T21:54:19.243 INFO:teuthology.orchestra.run.smithi112.stdout:1599 read 0x8d7fd0e thru 0x8d82e85 (0x3178 bytes) 2024-09-04T21:54:19.243 INFO:teuthology.orchestra.run.smithi112.stdout:1600 read 0x1f959d4 thru 0x1f9ba7d (0x60aa bytes) 2024-09-04T21:54:19.244 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:19.259 INFO:teuthology.orchestra.run.smithi112.stdout:1601 writesame 0xdcb7f09 thru 0xdcc30ba data_size 0x5(0xb1b2 bytes) 2024-09-04T21:54:19.259 INFO:teuthology.orchestra.run.smithi112.stdout:1602 write 0x1f218e4 thru 0x1f2324c (0x1969 bytes) 2024-09-04T21:54:19.261 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:19.261 INFO:teuthology.orchestra.run.smithi112.stdout:1605 punch from 0x72fdea to 0x73a6ca, (0xa8e0 bytes) 2024-09-04T21:54:19.261 INFO:teuthology.orchestra.run.smithi112.stdout:1606 read 0x8f8ca thru 0x93dce (0x4505 bytes) 2024-09-04T21:54:19.263 INFO:teuthology.orchestra.run.smithi112.stdout:1607 write 0x5ac9e94 thru 0x5ad8471 (0xe5de bytes) 2024-09-04T21:54:19.266 INFO:teuthology.orchestra.run.smithi112.stdout:1608 write 0x456814a thru 0x45778f1 (0xf7a8 bytes) 2024-09-04T21:54:19.269 INFO:teuthology.orchestra.run.smithi112.stdout:1610 read 0xb30355a thru 0xb30cf34 (0x99db bytes) 2024-09-04T21:54:19.269 INFO:teuthology.orchestra.run.smithi112.stdout:1611 write 0x1503133 thru 0x15093fa (0x62c8 bytes) 2024-09-04T21:54:19.273 INFO:teuthology.orchestra.run.smithi112.stdout:1612 clone 9 order 21 su 16384 sc 15 2024-09-04T21:54:19.330 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:19.330+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:19.411 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:19.411 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:19.414 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:19.411+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:19.427 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:19.515 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:19.514+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:54:19.598 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:19.599 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:19.614 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:19.616 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:19.615+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:19.665 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone8 from 0xdcc30bb (overlap 0x248e089) to 0x15eed1f 2024-09-04T21:54:19.717 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:19.715+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:19.776 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:19.776 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:19.789 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:19.818 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:19.815+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:19.908 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-09-04T21:54:19.908 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:19.908 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:19.919 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:19.915+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:19.924 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:19.954 INFO:teuthology.orchestra.run.smithi112.stdout:1614 writesame 0xc74db9a thru 0xc75ca78 data_size 0x1(0xeedf bytes) 2024-09-04T21:54:19.999 INFO:teuthology.orchestra.run.smithi112.stdout:1616 write 0x37673db thru 0x37770e1 (0xfd07 bytes) 2024-09-04T21:54:20.019 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:20.018+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:20.057 INFO:teuthology.orchestra.run.smithi112.stdout:1618 write 0x84331f4 thru 0x8435dde (0x2beb bytes) 2024-09-04T21:54:20.068 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:20.068 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:20.073 INFO:teuthology.orchestra.run.smithi112.stdout:1619 read 0x6bc7026 thru 0x6bd663c (0xf617 bytes) 2024-09-04T21:54:20.075 INFO:teuthology.orchestra.run.smithi112.stdout:1620 write 0x8fba372 thru 0x8fc4317 (0x9fa6 bytes) 2024-09-04T21:54:20.083 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:20.104 INFO:teuthology.orchestra.run.smithi112.stdout:1621 writesame 0x564f8c3 thru 0x564fdcd data_size 0x1(0x50b bytes) 2024-09-04T21:54:20.120 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:20.119+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:20.125 INFO:teuthology.orchestra.run.smithi112.stdout:1623 writesame 0x7f4f8fa thru 0x7f5795b data_size 0x1(0x8062 bytes) 2024-09-04T21:54:20.142 INFO:teuthology.orchestra.run.smithi112.stdout:1624 punch from 0xd200836 to 0xd208501, (0x7ccb bytes) 2024-09-04T21:54:20.143 INFO:teuthology.orchestra.run.smithi112.stdout:1625 write 0xfc7914 thru 0xfd35fb (0xbce8 bytes) 2024-09-04T21:54:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:1627 write 0x97d7742 thru 0x97d8fb8 (0x1877 bytes) 2024-09-04T21:54:20.205 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:20.205 INFO:teuthology.orchestra.run.smithi112.stdout:1629 punch from 0x1a43ae7 to 0x1a46064, (0x257d bytes) 2024-09-04T21:54:20.205 INFO:teuthology.orchestra.run.smithi112.stdout:1630 read 0x65abc42 thru 0x65b59ec (0x9dab bytes) 2024-09-04T21:54:20.205 INFO:teuthology.orchestra.run.smithi112.stdout:1631 write 0xcef47d6 thru 0xcefde63 (0x968e bytes) 2024-09-04T21:54:20.221 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:20.219+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:20.222 INFO:teuthology.orchestra.run.smithi112.stdout:1632 read 0x54201ca thru 0x542bbf0 (0xba27 bytes) 2024-09-04T21:54:20.223 INFO:teuthology.orchestra.run.smithi112.stdout:1633 write 0x4f0b7c9 thru 0x4f18921 (0xd159 bytes) 2024-09-04T21:54:20.261 INFO:teuthology.orchestra.run.smithi112.stdout:1634 write 0x2f2c2c3 thru 0x2f2cdbd (0xafb bytes) 2024-09-04T21:54:20.275 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:20.275 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:20.280 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:20.280 INFO:teuthology.orchestra.run.smithi112.stdout:1637 write 0x7c14243 thru 0x7c19e5c (0x5c1a bytes) 2024-09-04T21:54:20.290 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:20.291 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:20.291 INFO:teuthology.orchestra.run.smithi112.stdout:1639 write 0xb73b858 thru 0xb747901 (0xc0aa bytes) 2024-09-04T21:54:20.322 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:20.322+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:20.324 INFO:teuthology.orchestra.run.smithi112.stdout:1641 trunc from 0xdcc30bb to 0x67a7da1 2024-09-04T21:54:20.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:20.423 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:20.422+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:20.428 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:20.428 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:20.444 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:20.529 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:20.529 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:20.529 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:20.527+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:20.546 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:20.546 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:20.564 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:20.564 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:20.584 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:20.584 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:20.607 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:20.607 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:20.611 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:20.612 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:20.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:20.626 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:20.622+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:20.628 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:20.629 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:20.651 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:20.655 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:20.685 INFO:teuthology.orchestra.run.smithi112.stdout:1642 trunc from 0x67a7da1 to 0x45f1f62 2024-09-04T21:54:20.725 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:20.723+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:20.779 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:20.779 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:20.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:20.779 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:20.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.779 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.104093438, 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:20.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:20.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.104066653, 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:20.784 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.104037720000001, 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.789 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:20.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.104014073, 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103994291999999, 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103974897000001, 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103955623999999, 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103936402, 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103917565, 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103898162, 2024-09-04T21:54:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103877106000001, 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103844928000001, 2024-09-04T21:54:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103818707, 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103790498, 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:20.846 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103770458, 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103751076, 2024-09-04T21:54:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103727488000001, 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103707171, 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.861 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:20.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:20.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.103687619, 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.866 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:20.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:20.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.10366786, 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:20.871 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:20.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:20.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:20.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:20.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:20.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:20.876 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:20.878 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:20.878 INFO:teuthology.orchestra.run.smithi112.stdout:1643 writesame 0x1057227 thru 0x1060ed6 data_size 0x2e(0x9cb0 bytes) 2024-09-04T21:54:20.878 INFO:teuthology.orchestra.run.smithi112.stdout:1645 read 0x41182fd thru 0x4122586 (0xa28a bytes) 2024-09-04T21:54:20.878 INFO:teuthology.orchestra.run.smithi112.stdout:1647 punch from 0x3c4a0c2 to 0x3c50854, (0x6792 bytes) 2024-09-04T21:54:20.879 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:20.874+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:20.889 INFO:teuthology.orchestra.run.smithi112.stdout:1653 write 0xb7c530a thru 0xb7d0d0d (0xba04 bytes) 2024-09-04T21:54:20.901 INFO:teuthology.orchestra.run.smithi112.stdout:1654 read 0x8f4bc66 thru 0x8f5a4e9 (0xe884 bytes) 2024-09-04T21:54:20.901 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:20.901 INFO:teuthology.orchestra.run.smithi112.stdout:1656 writesame 0x4fe9737 thru 0x4feec95 data_size 0x1(0x555f bytes) 2024-09-04T21:54:20.910 INFO:teuthology.orchestra.run.smithi112.stdout:1657 read 0x3c13be6 thru 0x3c1c4ed (0x8908 bytes) 2024-09-04T21:54:20.913 INFO:teuthology.orchestra.run.smithi112.stdout:1658 trunc from 0xb7d0d0e to 0xc427d4c 2024-09-04T21:54:20.920 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:20.920 INFO:teuthology.orchestra.run.smithi112.stdout:1660 write 0x7fdef00 thru 0x7fe9931 (0xaa32 bytes) 2024-09-04T21:54:20.929 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:20.929 INFO:teuthology.orchestra.run.smithi112.stdout:1662 write 0x9ca1b74 thru 0x9ca7921 (0x5dae bytes) 2024-09-04T21:54:20.936 INFO:teuthology.orchestra.run.smithi112.stdout:1663 read 0x47c1ff2 thru 0x47c8113 (0x6122 bytes) 2024-09-04T21:54:20.937 INFO:teuthology.orchestra.run.smithi112.stdout:1664 punch from 0x179dc0c to 0x17a9792, (0xbb86 bytes) 2024-09-04T21:54:20.937 INFO:teuthology.orchestra.run.smithi112.stdout:1669 write 0x2b87f83 thru 0x2b8bf97 (0x4015 bytes) 2024-09-04T21:54:20.970 INFO:teuthology.orchestra.run.smithi112.stdout:1670 read 0xa16b6ff thru 0xa174b68 (0x946a bytes) 2024-09-04T21:54:20.978 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:20.975+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:20.979 INFO:teuthology.orchestra.run.smithi112.stdout:1671 writesame 0xd5d6e11 thru 0xd5dd2c0 data_size 0x48(0x64b0 bytes) 2024-09-04T21:54:20.979 INFO:teuthology.orchestra.run.smithi112.stdout:1672 read 0xd150017 thru 0xd150160 (0x14a bytes) 2024-09-04T21:54:20.979 INFO:teuthology.orchestra.run.smithi112.stdout:1673 write 0x8da1501 thru 0x8da40b2 (0x2bb2 bytes) 2024-09-04T21:54:20.995 INFO:teuthology.orchestra.run.smithi112.stdout:1674 trunc from 0xd5dd2c1 to 0x6b90e3e 2024-09-04T21:54:21.020 INFO:teuthology.orchestra.run.smithi112.stdout:1675 write 0x6bb269b thru 0x6bc07f3 (0xe159 bytes) 2024-09-04T21:54:21.028 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:21.029 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:21.033 INFO:teuthology.orchestra.run.smithi112.stdout:1676 write 0xce47b2 thru 0xceaba3 (0x63f2 bytes) 2024-09-04T21:54:21.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:21.071 INFO:teuthology.orchestra.run.smithi112.stdout:1677 punch from 0x6e2319 to 0x6f1cd3, (0xf9ba bytes) 2024-09-04T21:54:21.071 INFO:teuthology.orchestra.run.smithi112.stdout:1678 write 0x68c213a thru 0x68cdd64 (0xbc2b bytes) 2024-09-04T21:54:21.078 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:21.078+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:21.092 INFO:teuthology.orchestra.run.smithi112.stdout:1679 write 0xb271128 thru 0xb275d4c (0x4c25 bytes) 2024-09-04T21:54:21.099 INFO:teuthology.orchestra.run.smithi112.stdout:1680 write 0x7ae9260 thru 0x7af894f (0xf6f0 bytes) 2024-09-04T21:54:21.110 INFO:teuthology.orchestra.run.smithi112.stdout:1681 read 0x7488641 thru 0x7490d75 (0x8735 bytes) 2024-09-04T21:54:21.110 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.110 INFO:teuthology.orchestra.run.smithi112.stdout:1684 write 0x204b236 thru 0x2050727 (0x54f2 bytes) 2024-09-04T21:54:21.144 INFO:teuthology.orchestra.run.smithi112.stdout:1685 read 0x3166f3 thru 0x31f193 (0x8aa1 bytes) 2024-09-04T21:54:21.146 INFO:teuthology.orchestra.run.smithi112.stdout:1686 write 0x1db08a9 thru 0x1dc01ba (0xf912 bytes) 2024-09-04T21:54:21.172 INFO:teuthology.orchestra.run.smithi112.stdout:1687 punch from 0x5aef88b to 0x5afb51c, (0xbc91 bytes) 2024-09-04T21:54:21.179 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:21.175+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:21.181 INFO:teuthology.orchestra.run.smithi112.stdout:1688 write 0xbd18b29 thru 0xbd1c6a1 (0x3b79 bytes) 2024-09-04T21:54:21.188 INFO:teuthology.orchestra.run.smithi112.stdout:1689 punch from 0x8924911 to 0x89253fb, (0xaea bytes) 2024-09-04T21:54:21.188 INFO:teuthology.orchestra.run.smithi112.stdout:1690 read 0x15f8571 thru 0x1608163 (0xfbf3 bytes) 2024-09-04T21:54:21.200 INFO:teuthology.orchestra.run.smithi112.stdout:1691 write 0xce89481 thru 0xce92585 (0x9105 bytes) 2024-09-04T21:54:21.207 INFO:teuthology.orchestra.run.smithi112.stdout:1693 write 0x2883994 thru 0x2891a14 (0xe081 bytes) 2024-09-04T21:54:21.210 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:21.210 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:21.210 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:21.211 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:21.211 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:21.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:21.247 INFO:teuthology.orchestra.run.smithi112.stdout:1694 trunc from 0xce92586 to 0xcbb8767 2024-09-04T21:54:21.256 INFO:teuthology.orchestra.run.smithi112.stdout:1695 read 0x9a80321 thru 0x9a835b3 (0x3293 bytes) 2024-09-04T21:54:21.266 INFO:teuthology.orchestra.run.smithi112.stdout:1696 trunc from 0xcbb8767 to 0x5a7c222 2024-09-04T21:54:21.280 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:21.275+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:21.290 INFO:teuthology.orchestra.run.smithi112.stdout:1697 write 0x1ee7b9a thru 0x1ef1848 (0x9caf bytes) 2024-09-04T21:54:21.293 INFO:teuthology.orchestra.run.smithi112.stdout:1699 writesame 0x1de60e0 thru 0x1df44d1 data_size 0x2(0xe3f2 bytes) 2024-09-04T21:54:21.351 INFO:teuthology.orchestra.run.smithi112.stdout:1700 write 0x75466ff thru 0x754b8df (0x51e1 bytes) 2024-09-04T21:54:21.375 INFO:teuthology.orchestra.run.smithi112.stdout:1701 writesame 0xdb930a3 thru 0xdb9877d data_size 0x1(0x56db bytes) 2024-09-04T21:54:21.375 INFO:teuthology.orchestra.run.smithi112.stdout:1702 write 0x9061dd8 thru 0x9064526 (0x274f bytes) 2024-09-04T21:54:21.380 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:21.379+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:21.383 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:21.383 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:21.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:21.383 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:21.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.530974684, 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.384 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.404927421, 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:21.390 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:21.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:21.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.208373782999999, 2024-09-04T21:54:21.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:21.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:21.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:21.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.180984285000001, 2024-09-04T21:54:21.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:21.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.406 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.407 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.407 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.157719814, 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:21.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.416 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.129962024000001, 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:21.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:21.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:21.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.129846462, 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:21.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:21.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:21.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.829671677, 2024-09-04T21:54:21.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:21.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:21.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.435 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.670539124999999, 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:21.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:21.441 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.670313067, 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.442 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:21.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.193099998999999, 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.453 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.963711888000001, 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.454 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:21.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:21.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.665521247000001, 2024-09-04T21:54:21.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:21.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:21.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:21.466 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.665408392, 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:21.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:21.472 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.655747423999999, 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:21.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.635897699000001, 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:21.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:21.481 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.485669013000001, 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:21.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:21.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:21.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 12.819342108000001, 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:21.487 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:21.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:21.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 12.816202922, 2024-09-04T21:54:21.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 12.797891932000001, 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:21.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.501 INFO:teuthology.orchestra.run.smithi112.stdout:1703 write 0xc81ae5 thru 0xc8f2d6 (0xd7f2 bytes) 2024-09-04T21:54:21.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:21.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:21.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:21.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:21.503 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1704 read 0xa0d559f thru 0xa0d7907 (0x2369 bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1705 read 0xa56ee4c thru 0xa570586 (0x173b bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1706 write 0xbf9d0ac thru 0xbfac38d (0xf2e2 bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1709 read 0xad5448c thru 0xad5c6c9 (0x823e bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1710 read 0x589d670 thru 0x589f6f0 (0x2081 bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1712 punch from 0xc105361 to 0xc1146d2, (0xf371 bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1713 read 0x4b36a7d thru 0x4b3f99c (0x8f20 bytes) 2024-09-04T21:54:21.504 INFO:teuthology.orchestra.run.smithi112.stdout:1714 writesame 0xc0ab9d2 thru 0xc0b9914 data_size 0x73(0xdf43 bytes) 2024-09-04T21:54:21.506 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1715 punch from 0xb8d17b5 to 0xb8d26b6, (0xf01 bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1716 write 0x5652a9e thru 0x5655c2a (0x318d bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1717 writesame 0x4a5c405 thru 0x4a678d1 data_size 0x1(0xb4cd bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1722 write 0xca207b0 thru 0xca2c597 (0xbde8 bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1723 read 0x74a242a thru 0x74a386f (0x1446 bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1724 read 0xc0bc315 thru 0xc0cc2cb (0xffb7 bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1725 writesame 0x2800ce thru 0x2813fe data_size 0x1(0x1331 bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1727 read 0x8eab86a thru 0x8eb3552 (0x7ce9 bytes) 2024-09-04T21:54:21.509 INFO:teuthology.orchestra.run.smithi112.stdout:1730 read 0x6c154fa thru 0x6c1fef5 (0xa9fc bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1731 read 0x3647888 thru 0x364aa93 (0x320c bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1732 punch from 0xa8ecb91 to 0xa8f7b91, (0xb000 bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1733 writesame 0x8c1e642 thru 0x8c27b57 data_size 0x3(0x9516 bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1734 write 0x3bb362c thru 0x3bb7b62 (0x4537 bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1735 write 0xdb26643 thru 0xdb2cbee (0x65ac bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1736 write 0x41d9dea thru 0x41dd158 (0x336f bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1739 punch from 0xa725a1a to 0xa734f3e, (0xf524 bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1740 writesame 0x7a92bb4 thru 0x7aa094d data_size 0x1f(0xdd9a bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1741 punch from 0xc944a1b to 0xc94b2a2, (0x6887 bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1744 write 0x7a855a1 thru 0x7a88c06 (0x3666 bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1745 write 0x2dcc8f5 thru 0x2dd91d3 (0xc8df bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1746 write 0xcd288e8 thru 0xcd2f071 (0x678a bytes) 2024-09-04T21:54:21.510 INFO:teuthology.orchestra.run.smithi112.stdout:1747 trunc from 0xdb9877e to 0xd3f335d 2024-09-04T21:54:21.511 INFO:teuthology.orchestra.run.smithi112.stdout:1748 read 0x3b35933 thru 0x3b42c87 (0xd355 bytes) 2024-09-04T21:54:21.511 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:21.503+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:21.511 INFO:teuthology.orchestra.run.smithi112.stdout:1750 write 0x21bc41c thru 0x21caa98 (0xe67d bytes) 2024-09-04T21:54:21.539 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.540 INFO:teuthology.orchestra.run.smithi112.stdout:1752 punch from 0x9548df9 to 0x954d812, (0x4a19 bytes) 2024-09-04T21:54:21.540 INFO:teuthology.orchestra.run.smithi112.stdout:1753 write 0xbeffa77 thru 0xbf0addc (0xb366 bytes) 2024-09-04T21:54:21.543 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:54:21.547 INFO:teuthology.orchestra.run.smithi112.stdout:1755 punch from 0x8d0e95 to 0x8d8ac9, (0x7c34 bytes) 2024-09-04T21:54:21.563 INFO:teuthology.orchestra.run.smithi112.stdout:1757 trunc from 0xd3f335d to 0x4b5181e 2024-09-04T21:54:21.608 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:21.607+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:21.616 INFO:teuthology.orchestra.run.smithi112.stdout:1758 write 0x9beb114 thru 0x9bedc15 (0x2b02 bytes) 2024-09-04T21:54:21.622 INFO:teuthology.orchestra.run.smithi112.stdout:1759 read 0x14759bc thru 0x1477433 (0x1a78 bytes) 2024-09-04T21:54:21.623 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.637 INFO:teuthology.orchestra.run.smithi112.stdout:1762 write 0xdb43eb3 thru 0xdb50841 (0xc98f bytes) 2024-09-04T21:54:21.647 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.647 INFO:teuthology.orchestra.run.smithi112.stdout:1765 punch from 0x64a7c23 to 0x64b51b6, (0xd593 bytes) 2024-09-04T21:54:21.647 INFO:teuthology.orchestra.run.smithi112.stdout:1766 write 0x902a703 thru 0x903957e (0xee7c bytes) 2024-09-04T21:54:21.659 INFO:teuthology.orchestra.run.smithi112.stdout:1768 punch from 0xd2218a6 to 0xd22ee86, (0xd5e0 bytes) 2024-09-04T21:54:21.660 INFO:teuthology.orchestra.run.smithi112.stdout:1769 read 0x362354a thru 0x362e4c6 (0xaf7d bytes) 2024-09-04T21:54:21.660 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:21.661 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:21.661 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:21.661 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:21.661 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.661 INFO:teuthology.orchestra.run.smithi112.stdout:1771 read 0x49c5fd thru 0x4a48f3 (0x82f7 bytes) 2024-09-04T21:54:21.662 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.669 INFO:teuthology.orchestra.run.smithi112.stdout:1774 writesame 0xe23c2e8 thru 0xe248be5 data_size 0xd(0xc8fe bytes) 2024-09-04T21:54:21.670 INFO:teuthology.orchestra.run.smithi112.stdout:1775 write 0x7b8236b thru 0x7b88215 (0x5eab bytes) 2024-09-04T21:54:21.677 INFO:teuthology.orchestra.run.smithi112.stdout:1776 read 0x6b6f294 thru 0x6b72e0c (0x3b79 bytes) 2024-09-04T21:54:21.677 INFO:teuthology.orchestra.run.smithi112.stdout:1777 read 0x3b3d77c thru 0x3b47f70 (0xa7f5 bytes) 2024-09-04T21:54:21.678 INFO:teuthology.orchestra.run.smithi112.stdout:1778 read 0x98f5e78 thru 0x98fcd7f (0x6f08 bytes) 2024-09-04T21:54:21.678 INFO:teuthology.orchestra.run.smithi112.stdout:1779 read 0x370c24b thru 0x3711424 (0x51da bytes) 2024-09-04T21:54:21.679 INFO:teuthology.orchestra.run.smithi112.stdout:1780 read 0x9d90328 thru 0x9d92236 (0x1f0f bytes) 2024-09-04T21:54:21.680 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:21.698 INFO:teuthology.orchestra.run.smithi112.stdout:1781 trunc from 0xe248be6 to 0x173bb0e 2024-09-04T21:54:21.744 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:21.743+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:21.777 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.796 INFO:teuthology.orchestra.run.smithi112.stdout:1783 write 0x9a274e0 thru 0x9a33d40 (0xc861 bytes) 2024-09-04T21:54:21.811 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:21.807+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:21.814 INFO:teuthology.orchestra.run.smithi112.stdout:1784 trunc from 0x9a33d41 to 0x48e878e 2024-09-04T21:54:21.830 INFO:teuthology.orchestra.run.smithi112.stdout:1786 read 0x44bbb8a thru 0x44c231f (0x6796 bytes) 2024-09-04T21:54:21.843 INFO:teuthology.orchestra.run.smithi112.stdout:1787 write 0x7e15c5c thru 0x7e21091 (0xb436 bytes) 2024-09-04T21:54:21.843 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:21.843 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:21.843 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:21.843 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:21.843 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:21.851 INFO:teuthology.orchestra.run.smithi112.stdout:1789 write 0x709c32 thru 0x714c81 (0xb050 bytes) 2024-09-04T21:54:21.855 INFO:teuthology.orchestra.run.smithi112.stdout:1790 read 0x106c56 thru 0x10add8 (0x4183 bytes) 2024-09-04T21:54:21.857 INFO:teuthology.orchestra.run.smithi112.stdout:1791 write 0x3414e1f thru 0x34200d3 (0xb2b5 bytes) 2024-09-04T21:54:21.861 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:21.866 INFO:teuthology.orchestra.run.smithi112.stdout:1792 punch from 0xe2773b to 0xe30301, (0x8bc6 bytes) 2024-09-04T21:54:21.866 INFO:teuthology.orchestra.run.smithi112.stdout:1793 read 0x59aa3d4 thru 0x59b3b0e (0x973b bytes) 2024-09-04T21:54:21.881 INFO:teuthology.orchestra.run.smithi112.stdout:1794 write 0xc2dfb0c thru 0xc2eebe3 (0xf0d8 bytes) 2024-09-04T21:54:21.892 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.892 INFO:teuthology.orchestra.run.smithi112.stdout:1797 writesame 0x60aeb72 thru 0x60b1bda data_size 0x33(0x3069 bytes) 2024-09-04T21:54:21.910 INFO:teuthology.orchestra.run.smithi112.stdout:1798 writesame 0xecfc9b6 thru 0xed046b5 data_size 0x19(0x7d00 bytes) 2024-09-04T21:54:21.910 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.911 INFO:teuthology.orchestra.run.smithi112.stdout:1800 punch from 0xb115d34 to 0xb119595, (0x3861 bytes) 2024-09-04T21:54:21.911 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.911 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.911 INFO:teuthology.orchestra.run.smithi112.stdout:1803 read 0x55fc438 thru 0x5600803 (0x43cc bytes) 2024-09-04T21:54:21.912 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:21.910+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:21.925 INFO:teuthology.orchestra.run.smithi112.stdout:1804 trunc from 0xed046b6 to 0x30c82dd 2024-09-04T21:54:21.946 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.946 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:21.946 INFO:teuthology.orchestra.run.smithi112.stdout:1807 read 0x15193da thru 0x151b341 (0x1f68 bytes) 2024-09-04T21:54:21.947 INFO:teuthology.orchestra.run.smithi112.stdout:1808 punch from 0x2b62721 to 0x2b6d8aa, (0xb189 bytes) 2024-09-04T21:54:21.952 INFO:teuthology.orchestra.run.smithi112.stdout:1809 trunc from 0x30c82dd to 0x7a5e56c 2024-09-04T21:54:21.959 INFO:teuthology.orchestra.run.smithi112.stdout:1810 read 0x60595cb thru 0x6066e38 (0xd86e bytes) 2024-09-04T21:54:21.959 INFO:teuthology.orchestra.run.smithi112.stdout:1811 write 0x2cf6810 thru 0x2cfd3b1 (0x6ba2 bytes) 2024-09-04T21:54:21.967 INFO:teuthology.orchestra.run.smithi112.stdout:1812 read 0xde009d thru 0xde3232 (0x3196 bytes) 2024-09-04T21:54:21.977 INFO:teuthology.orchestra.run.smithi112.stdout:1814 trunc from 0x7a5e56c to 0x161954f 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1421 3.0 e28/22 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head v 28'14, mlcod=27'12)", 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.979684+0000", 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.015491892, 2024-09-04T21:54:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025395589999999998, 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979684+0000", 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979684+0000", 2024-09-04T21:54:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979686+0000", 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0660000000000002e-06 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980465+0000", 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00077882400000000001 2024-09-04T21:54:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980505+0000", 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9603e-05 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980524+0000", 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8736000000000002e-05 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980678+0000", 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015410200000000001 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.980701+0000", 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3754999999999999e-05 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982122+0000", 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001420457 2024-09-04T21:54:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982224+0000", 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010201599999999999 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1555 3.4 e28/22 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head v 28'17, mlcod=28'15)", 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.225928+0000", 2024-09-04T21:54:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.769248435000002, 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002570541, 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225928+0000", 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225928+0000", 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225931+0000", 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7669999999999999e-06 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225963+0000", 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2095999999999999e-05 2024-09-04T21:54:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226000+0000", 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7299000000000003e-05 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226022+0000", 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2459999999999998e-05 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226211+0000", 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000188655 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.226279+0000", 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8003000000000006e-05 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228381+0000", 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021015919999999998 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228498+0000", 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117669 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1560 3.4 e28/22 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head v 28'18, mlcod=28'16)", 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.253857+0000", 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.741319426, 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025741710000000001, 2024-09-04T21:54:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253857+0000", 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253857+0000", 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253860+0000", 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9730000000000002e-06 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253877+0000", 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7353000000000001e-05 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253921+0000", 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3878999999999998e-05 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253962+0000", 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0763999999999998e-05 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254124+0000", 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000162282 2024-09-04T21:54:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254202+0000", 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8262000000000004e-05 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256280+0000", 2024-09-04T21:54:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020774119999999998 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256431+0000", 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015124600000000001 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7156 3.4 e30/22 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head v 30'66, mlcod=30'64)", 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.282130699, 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002957763, 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713047+0000", 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.528e-06 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713248+0000", 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201287 2024-09-04T21:54:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713277+0000", 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8487e-05 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713288+0000", 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0832000000000001e-05 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713359+0000", 2024-09-04T21:54:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0993000000000003e-05 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713382+0000", 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3580000000000001e-05 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715959+0000", 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025766869999999998 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716003+0000", 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4369000000000002e-05 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7159 3.9 e30/22 3:93c2ae1d:::rbd_data.118867f9efa3.000000000000000f:head v 30'14, mlcod=29'10)", 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.279961085, 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031393580000000001, 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.015 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715216+0000", 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6300000000000004e-07 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715223+0000", 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2980000000000003e-06 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715237+0000", 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3361e-05 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715244+0000", 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.362e-06 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716436+0000", 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00119187 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716456+0000", 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0562999999999999e-05 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718316+0000", 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018595720000000001 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718354+0000", 2024-09-04T21:54:22.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8469e-05 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7729 3.4 e31/22 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head v 31'74, mlcod=30'72)", 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.096615965, 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031016809999999998, 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.020 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898561+0000", 2024-09-04T21:54:22.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898620+0000", 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8813000000000001e-05 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898654+0000", 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4292999999999998e-05 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898675+0000", 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1024000000000001e-05 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898834+0000", 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015876599999999999 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898899+0000", 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5209000000000005e-05 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901577+0000", 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026777849999999998 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901662+0000", 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4983999999999998e-05 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7863 3.9 e32/22 3:9478b05c:::rbd_data.1188b1294ada.000000000000005b:head v 32'21, mlcod=31'18)", 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.430545275, 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002738262, 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564632+0000", 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4000000000000001e-07 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564643+0000", 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0966e-05 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564691+0000", 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8231e-05 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564714+0000", 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2857000000000002e-05 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.565011+0000", 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029731499999999998 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.565062+0000", 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1258000000000003e-05 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567253+0000", 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021909210000000002 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567369+0000", 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000115974 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7862 3.4 e32/22 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head v 32'76, mlcod=30'73)", 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.429077713, 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031099529999999999, 2024-09-04T21:54:22.029 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566099+0000", 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9999999999999996e-07 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566110+0000", 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1009000000000001e-05 2024-09-04T21:54:22.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566141+0000", 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0244e-05 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566161+0000", 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0224000000000002e-05 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566407+0000", 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024641000000000002 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566463+0000", 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6113999999999998e-05 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569116+0000", 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026528519999999998 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569208+0000", 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2299999999999994e-05 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14326 3.0 3:0619fb3a:::rbd_header.118816801d55:head [watch watch cookie 139952114223744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.3728357100000004, 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00266585, 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.034 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14326 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622342+0000", 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622350+0000", 2024-09-04T21:54:22.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0190000000000006e-06 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622373+0000", 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2634999999999999e-05 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622385+0000", 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622460+0000", 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4231000000000001e-05 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622524+0000", 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4548000000000006e-05 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622642+0000", 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117707 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.623541+0000", 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00089878600000000005 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.624756+0000", 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012156300000000001 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.624853+0000", 2024-09-04T21:54:22.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6292000000000006e-05 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.625006+0000", 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015378000000000001 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14565 3.0 3:05912ce2:::rbd_data.118816801d55.00000000000000ee:head [sparse-read 0~524288] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:22.040 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.27344934, 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030052669999999998, 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14565 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721728+0000", 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0669999999999999e-06 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721733+0000", 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0189999999999997e-06 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721749+0000", 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6303e-05 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721757+0000", 2024-09-04T21:54:22.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7680000000000008e-06 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723062+0000", 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001305283 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.724732+0000", 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001669827 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14711 3.0 3:0619fb3a:::rbd_header.118816801d55:head [watch unwatch cookie 139952114223744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1761694909999996, 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028282910000000001, 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14711 2024-09-04T21:54:22.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819010+0000", 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8569999999999999e-06 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819020+0000", 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0808999999999999e-05 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819049+0000", 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8697999999999999e-05 2024-09-04T21:54:22.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819061+0000", 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1617000000000001e-05 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819171+0000", 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011018000000000001 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819302+0000", 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000130751 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819493+0000", 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019150100000000001 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.820539+0000", 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010458620000000001 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821701+0000", 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011623009999999999 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821758+0000", 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6988999999999999e-05 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821835+0000", 2024-09-04T21:54:22.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6725999999999997e-05 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:21464 3.4 3:264f9539:::rbd_data.118892c64562.000000000000004f:head [set-alloc-hint object_size 2097152 write_size 2097152,write 465780~9356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.935034+0000", 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.060142358, 2024-09-04T21:54:22.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030008629999999999, 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 21464 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935034+0000", 2024-09-04T21:54:22.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935034+0000", 2024-09-04T21:54:22.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935036+0000", 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.159e-06 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935049+0000", 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2575000000000001e-05 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935068+0000", 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9405000000000001e-05 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935078+0000", 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0277e-05 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935153+0000", 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5220000000000001e-05 2024-09-04T21:54:22.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935283+0000", 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012953699999999999 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935385+0000", 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010186799999999999 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.936350+0000", 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00096469600000000002 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.937960+0000", 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001610301 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.938002+0000", 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2364000000000003e-05 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.938035+0000", 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2461e-05 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:21751 3.4 3:264f9539:::rbd_data.118892c64562.000000000000004f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.001161+0000", 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.99401551200000005, 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038266440000000001, 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.059 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 21751 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001161+0000", 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001161+0000", 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001163+0000", 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4250000000000001e-06 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001171+0000", 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9300000000000003e-06 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001193+0000", 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2103000000000002e-05 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001204+0000", 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0969e-05 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001280+0000", 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6143000000000006e-05 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001378+0000", 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7761000000000001e-05 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001496+0000", 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117997 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.002109+0000", 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000612618 2024-09-04T21:54:22.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.004887+0000", 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027781720000000002 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.004943+0000", 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6069999999999997e-05 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.004987+0000", 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4456000000000003e-05 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23212 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.316313+0000", 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.67886330900000003, 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.012575382, 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.065 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23212 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.316313+0000", 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.316313+0000", 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.316315+0000", 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9589999999999998e-06 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317607+0000", 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012921689999999999 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317629+0000", 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.215e-05 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317638+0000", 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8380000000000008e-06 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317713+0000", 2024-09-04T21:54:22.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5310999999999998e-05 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317852+0000", 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013871399999999999 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.318878+0000", 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010260499999999999 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.322933+0000", 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040546710000000001 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.328809+0000", 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058764170000000001 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.328853+0000", 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4162999999999997e-05 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.328888+0000", 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4940000000000001e-05 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23244 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [stat,write 868352~13015] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.385214+0000", 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.60996181800000004, 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031555229999999999, 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23244 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385214+0000", 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385214+0000", 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385216+0000", 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.655e-06 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385229+0000", 2024-09-04T21:54:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.331e-05 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385242+0000", 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2906e-05 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385258+0000", 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5875e-05 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385324+0000", 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5587999999999996e-05 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385387+0000", 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3602000000000005e-05 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385472+0000", 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4652000000000002e-05 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.386310+0000", 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083819500000000002 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.388253+0000", 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019430610000000001 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.388321+0000", 2024-09-04T21:54:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7583000000000007e-05 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.388370+0000", 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9095999999999999e-05 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23301 3.0 3:04b5afe4:::rbd_data.118892c64562.000000000000006a:head [set-alloc-hint object_size 2097152 write_size 2097152,write 638976~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.465580+0000", 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.52959645700000002, 2024-09-04T21:54:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035173539999999999, 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23301 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465580+0000", 2024-09-04T21:54:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465580+0000", 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465582+0000", 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465594+0000", 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2432999999999999e-05 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465609+0000", 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4763000000000001e-05 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465615+0000", 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2179999999999996e-06 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465728+0000", 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011267 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465952+0000", 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022462099999999999 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.466117+0000", 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016514700000000001 2024-09-04T21:54:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.467064+0000", 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00094625899999999997 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.469009+0000", 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001945612 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.469059+0000", 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9796000000000002e-05 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.469097+0000", 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7991000000000003e-05 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23323 3.0 3:04b5afe4:::rbd_data.118892c64562.000000000000006a:head [truncate 147456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.496920+0000", 2024-09-04T21:54:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.49825609799999998, 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026234370000000002, 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23323 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496920+0000", 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496920+0000", 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496923+0000", 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0639999999999998e-06 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496933+0000", 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496955+0000", 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2583e-05 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496964+0000", 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7930000000000003e-06 2024-09-04T21:54:22.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.497078+0000", 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000114144 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.497203+0000", 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012433499999999999 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.497378+0000", 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017547099999999999 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.498251+0000", 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087273400000000001 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.499456+0000", 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012056809999999999 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.499491+0000", 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4390000000000001e-05 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.499544+0000", 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2741999999999997e-05 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23513 3.0 3:04b5afe4:::rbd_data.118892c64562.000000000000006a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.580224+0000", 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.41495260499999997, 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026765420000000001, 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23513 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.090 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580224+0000", 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580224+0000", 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580225+0000", 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6589999999999999e-06 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580234+0000", 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0829999999999993e-06 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580250+0000", 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6127e-05 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580259+0000", 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2579999999999995e-06 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580336+0000", 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7667e-05 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580434+0000", 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7844999999999996e-05 2024-09-04T21:54:22.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580571+0000", 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013706999999999999 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.581099+0000", 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00052805199999999999 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.582819+0000", 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017201320000000001 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.582855+0000", 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.54e-05 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.582900+0000", 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5249e-05 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23649 3.4 3:264f9539:::rbd_data.118892c64562.000000000000004f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.716257+0000", 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.27891950599999998, 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029682300000000001, 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.096 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23649 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716257+0000", 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716257+0000", 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716259+0000", 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9300000000000002e-06 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716265+0000", 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1360000000000002e-06 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716282+0000", 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6816e-05 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716290+0000", 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0790000000000001e-06 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716361+0000", 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0894000000000003e-05 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716451+0000", 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0403999999999997e-05 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716563+0000", 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011253500000000001 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.717130+0000", 2024-09-04T21:54:22.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056608899999999998 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.719109+0000", 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019795469999999999 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.719175+0000", 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5440999999999999e-05 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.719225+0000", 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0358999999999997e-05 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23918 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [stat,truncate 1622016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.726035+0000", 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.269141032, 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032032509999999998, 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.102 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23918 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726035+0000", 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726035+0000", 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726037+0000", 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0140000000000001e-06 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726045+0000", 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3209999999999999e-06 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726063+0000", 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8875000000000001e-05 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726073+0000", 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3409999999999992e-06 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726154+0000", 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1396999999999993e-05 2024-09-04T21:54:22.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726264+0000", 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010967300000000001 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726426+0000", 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000162275 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.727165+0000", 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00073903399999999996 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.729124+0000", 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019587480000000002 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.729191+0000", 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6913000000000001e-05 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.729238+0000", 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7660000000000001e-05 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.108 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:22.109 INFO:teuthology.orchestra.run.smithi112.stdout:1816 read 0x502df3 thru 0x509c07 (0x6e15 bytes) 2024-09-04T21:54:22.109 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.110 INFO:teuthology.orchestra.run.smithi112.stdout:1818 writesame 0x625620 thru 0x631ddb data_size 0x6(0xc7bc bytes) 2024-09-04T21:54:22.110 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.110 INFO:teuthology.orchestra.run.smithi112.stdout:1820 read 0x138d424 thru 0x139ba3b (0xe618 bytes) 2024-09-04T21:54:22.110 INFO:teuthology.orchestra.run.smithi112.stdout:1822 read 0x865dfb thru 0x86876d (0x2973 bytes) 2024-09-04T21:54:22.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:22.112 INFO:teuthology.orchestra.run.smithi112.stdout:1823 trunc from 0x161954f to 0xc82b452 2024-09-04T21:54:22.112 INFO:teuthology.orchestra.run.smithi112.stdout:1824 write 0xa1d8ea8 thru 0xa1e8ea5 (0xfffe bytes) 2024-09-04T21:54:22.112 INFO:teuthology.orchestra.run.smithi112.stdout:1825 write 0x5621918 thru 0x5628a55 (0x713e bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1826 punch from 0x51c9550 to 0x51d2981, (0x9431 bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1828 read 0x9217d41 thru 0x92267ce (0xea8e bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1830 read 0x33e7e56 thru 0x33ecfb3 (0x515e bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1832 trunc from 0xc82b452 to 0x574b595 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1834 read 0x1b1c0fe thru 0x1b2597c (0x987f bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1836 write 0x194a4a5 thru 0x1950c19 (0x6775 bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1837 read 0x446dec2 thru 0x447452c (0x666b bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1839 punch from 0x446600d to 0x447445e, (0xe451 bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1843 read 0x2bcbc0d thru 0x2bd207d (0x6471 bytes) 2024-09-04T21:54:22.113 INFO:teuthology.orchestra.run.smithi112.stdout:1846 trunc from 0x574b595 to 0x23bba9e 2024-09-04T21:54:22.114 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:22.107+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:22.137 INFO:teuthology.orchestra.run.smithi112.stdout:1847 write 0x8bb5283 thru 0x8bc2448 (0xd1c6 bytes) 2024-09-04T21:54:22.149 INFO:teuthology.orchestra.run.smithi112.stdout:1848 punch from 0x50659dd to 0x506fdd1, (0xa3f4 bytes) 2024-09-04T21:54:22.149 INFO:teuthology.orchestra.run.smithi112.stdout:1849 writesame 0x2196ca6 thru 0x219e63e data_size 0x1(0x7999 bytes) 2024-09-04T21:54:22.178 INFO:teuthology.orchestra.run.smithi112.stdout:1850 read 0x88e22d5 thru 0x88e34bd (0x11e9 bytes) 2024-09-04T21:54:22.178 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.187 INFO:teuthology.orchestra.run.smithi112.stdout:1852 trunc from 0x8bc2449 to 0xd6c4b46 2024-09-04T21:54:22.196 INFO:teuthology.orchestra.run.smithi112.stdout:1853 read 0xd1cf95c thru 0xd1dd9b2 (0xe057 bytes) 2024-09-04T21:54:22.197 INFO:teuthology.orchestra.run.smithi112.stdout:1855 write 0xc6614 thru 0xcda3a (0x7427 bytes) 2024-09-04T21:54:22.199 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.199 INFO:teuthology.orchestra.run.smithi112.stdout:1857 punch from 0x3f8f98e to 0x3f9f4ab, (0xfb1d bytes) 2024-09-04T21:54:22.199 INFO:teuthology.orchestra.run.smithi112.stdout:1858 read 0xbaf34fd thru 0xbafe7b8 (0xb2bc bytes) 2024-09-04T21:54:22.199 INFO:teuthology.orchestra.run.smithi112.stdout:1859 read 0x52e8b63 thru 0x52f84d9 (0xf977 bytes) 2024-09-04T21:54:22.199 INFO:teuthology.orchestra.run.smithi112.stdout:1860 clone 10 order 19 su 65536 sc 4 2024-09-04T21:54:22.210 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:22.207+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:22.261 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:22.262 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:22.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:22.316 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:22.315+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:22.412 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:22.411+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:22.441 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:22.441 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:22.441 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:22.441 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:22.441 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:22.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:22.514 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:54:22.515 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:22.511+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:22.614 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:22.610+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7369 3.2 3:473dc813:::rbd_data.1188b1294ada.0000000000000008:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:22.635 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.517988812, 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035947, 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.637 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7369 2024-09-04T21:54:22.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.637 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:22.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:22.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117106+0000", 2024-09-04T21:54:22.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9799999999999998e-06 2024-09-04T21:54:22.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117118+0000", 2024-09-04T21:54:22.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1792e-05 2024-09-04T21:54:22.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117149+0000", 2024-09-04T21:54:22.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0412999999999998e-05 2024-09-04T21:54:22.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117163+0000", 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4779e-05 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117282+0000", 2024-09-04T21:54:22.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011867 2024-09-04T21:54:22.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117441+0000", 2024-09-04T21:54:22.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015941300000000001 2024-09-04T21:54:22.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:22.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.123847+0000", 2024-09-04T21:54:22.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064053850000000004 2024-09-04T21:54:22.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:22.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152852+0000", 2024-09-04T21:54:22.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02900525 2024-09-04T21:54:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152883+0000", 2024-09-04T21:54:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0756999999999997e-05 2024-09-04T21:54:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152973+0000", 2024-09-04T21:54:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0002999999999999e-05 2024-09-04T21:54:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.153050+0000", 2024-09-04T21:54:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7558000000000005e-05 2024-09-04T21:54:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.650 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.650 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7714 3.7 3:e20eb1bd:::rbd_data.1188b1294ada.0000000000000030:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:22.650 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:22.650 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.803273753999999, 2024-09-04T21:54:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034647877000000001, 2024-09-04T21:54:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.652 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.652 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7714 2024-09-04T21:54:22.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.652 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:22.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831822+0000", 2024-09-04T21:54:22.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1599999999999998e-06 2024-09-04T21:54:22.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831833+0000", 2024-09-04T21:54:22.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1472e-05 2024-09-04T21:54:22.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831865+0000", 2024-09-04T21:54:22.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1707e-05 2024-09-04T21:54:22.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831880+0000", 2024-09-04T21:54:22.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4871e-05 2024-09-04T21:54:22.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.832016+0000", 2024-09-04T21:54:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000136685 2024-09-04T21:54:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.832254+0000", 2024-09-04T21:54:22.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023754099999999999 2024-09-04T21:54:22.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:22.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866177+0000", 2024-09-04T21:54:22.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.03392299 2024-09-04T21:54:22.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:22.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866313+0000", 2024-09-04T21:54:22.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135995 2024-09-04T21:54:22.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866466+0000", 2024-09-04T21:54:22.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000153456 2024-09-04T21:54:22.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20788 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 11993088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9134884680000002, 2024-09-04T21:54:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024856677000000001, 2024-09-04T21:54:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20788 2024-09-04T21:54:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:22.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2399999999999998e-07 2024-09-04T21:54:22.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721606+0000", 2024-09-04T21:54:22.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3010000000000001e-06 2024-09-04T21:54:22.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721613+0000", 2024-09-04T21:54:22.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3679999999999998e-06 2024-09-04T21:54:22.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721618+0000", 2024-09-04T21:54:22.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8729999999999998e-06 2024-09-04T21:54:22.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746259+0000", 2024-09-04T21:54:22.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024641237 2024-09-04T21:54:22.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746343+0000", 2024-09-04T21:54:22.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3745000000000005e-05 2024-09-04T21:54:22.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746460+0000", 2024-09-04T21:54:22.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011762899999999999 2024-09-04T21:54:22.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.673 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.674 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20789 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12156928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.674 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:22.675 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.913468315, 2024-09-04T21:54:22.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025041028, 2024-09-04T21:54:22.675 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.675 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.675 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.675 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.676 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.676 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.676 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20789 2024-09-04T21:54:22.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.676 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:22.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:22.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:22.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7300000000000002e-07 2024-09-04T21:54:22.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721626+0000", 2024-09-04T21:54:22.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7460000000000001e-06 2024-09-04T21:54:22.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721631+0000", 2024-09-04T21:54:22.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5230000000000003e-06 2024-09-04T21:54:22.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721636+0000", 2024-09-04T21:54:22.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.933e-06 2024-09-04T21:54:22.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746494+0000", 2024-09-04T21:54:22.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024857240999999999 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746575+0000", 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1833000000000001e-05 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746665+0000", 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9378999999999999e-05 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20790 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12320768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721645+0000", 2024-09-04T21:54:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.913446915, 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025214770000000001, 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20790 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721645+0000", 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721645+0000", 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721646+0000", 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8299999999999997e-07 2024-09-04T21:54:22.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721648+0000", 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3609999999999999e-06 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721655+0000", 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8730000000000001e-06 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721660+0000", 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1889999999999998e-06 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746696+0000", 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025036202 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746781+0000", 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4690000000000004e-05 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746860+0000", 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8972000000000002e-05 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20791 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12566528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9134245299999999, 2024-09-04T21:54:22.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025394371999999998, 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20791 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:22.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721674+0000", 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4250000000000003e-06 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721681+0000", 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9500000000000004e-06 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721688+0000", 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4420000000000003e-06 2024-09-04T21:54:22.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746892+0000", 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02520441 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746978+0000", 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5370000000000004e-05 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747062+0000", 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4321999999999997e-05 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20792 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12730368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721696+0000", 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9133961390000001, 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025565674, 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20792 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721696+0000", 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721696+0000", 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721697+0000", 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2799999999999995e-07 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721699+0000", 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5320000000000001e-06 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721706+0000", 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.759e-06 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721711+0000", 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4280000000000004e-06 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747092+0000", 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02538033 2024-09-04T21:54:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747178+0000", 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5801000000000001e-05 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747262+0000", 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4096000000000003e-05 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20793 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12894208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721718+0000", 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.913373794, 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025730287000000001, 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20793 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721718+0000", 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721718+0000", 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721719+0000", 2024-09-04T21:54:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-07 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721721+0000", 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2230000000000001e-06 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721727+0000", 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4139999999999999e-06 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721733+0000", 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747295+0000", 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025562135 2024-09-04T21:54:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747371+0000", 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5903999999999997e-05 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747449+0000", 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7955999999999997e-05 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20794 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9133526139999999, 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025923188, 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20794 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7e-07 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721743+0000", 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6469999999999998e-06 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721749+0000", 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6810000000000001e-06 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721755+0000", 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2549999999999997e-06 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747505+0000", 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025750287 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747581+0000", 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5967e-05 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747663+0000", 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1880999999999998e-05 2024-09-04T21:54:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20795 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9133303100000001, 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026093022, 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20795 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6800000000000001e-07 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721764+0000", 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721771+0000", 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6069999999999996e-06 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721776+0000", 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9030000000000003e-06 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747695+0000", 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025918773999999999 2024-09-04T21:54:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747777+0000", 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2089000000000006e-05 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747855+0000", 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8118e-05 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20796 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13631488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721796+0000", 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9132960809999999, 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026249567000000001, 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20796 2024-09-04T21:54:22.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721796+0000", 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721796+0000", 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721797+0000", 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2699999999999999e-07 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721799+0000", 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4229999999999999e-06 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721807+0000", 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721812+0000", 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5539999999999997e-06 2024-09-04T21:54:22.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747888+0000", 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026075528000000001 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747970+0000", 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2181000000000005e-05 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748046+0000", 2024-09-04T21:54:22.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5438000000000005e-05 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20797 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13877248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9132715980000001, 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026421463999999999, 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.719 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20797 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7899999999999998e-07 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721823+0000", 2024-09-04T21:54:22.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1780000000000002e-06 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721830+0000", 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5270000000000004e-06 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721839+0000", 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3479999999999993e-06 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748079+0000", 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026239357000000001 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748164+0000", 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4982999999999996e-05 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748242+0000", 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8492000000000007e-05 2024-09-04T21:54:22.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.724 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.724 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20798 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14041088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.724 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9132452849999999, 2024-09-04T21:54:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026571786999999999, 2024-09-04T21:54:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.726 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20798 2024-09-04T21:54:22.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.726 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:22.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:22.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:22.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6199999999999998e-07 2024-09-04T21:54:22.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721849+0000", 2024-09-04T21:54:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1430000000000001e-06 2024-09-04T21:54:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721856+0000", 2024-09-04T21:54:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6159999999999998e-06 2024-09-04T21:54:22.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721861+0000", 2024-09-04T21:54:22.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0420000000000002e-06 2024-09-04T21:54:22.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748273+0000", 2024-09-04T21:54:22.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026411453000000001 2024-09-04T21:54:22.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748352+0000", 2024-09-04T21:54:22.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9412000000000005e-05 2024-09-04T21:54:22.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748419+0000", 2024-09-04T21:54:22.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6659e-05 2024-09-04T21:54:22.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.735 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.735 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20799 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14204928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.735 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:22.736 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9132235660000001, 2024-09-04T21:54:22.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027941948000000001, 2024-09-04T21:54:22.736 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.736 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.736 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.737 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20799 2024-09-04T21:54:22.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.738 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:22.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:22.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:22.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.82e-07 2024-09-04T21:54:22.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721871+0000", 2024-09-04T21:54:22.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1859999999999999e-06 2024-09-04T21:54:22.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721878+0000", 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6329999999999999e-06 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721883+0000", 2024-09-04T21:54:22.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0459999999999996e-06 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748452+0000", 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026568852 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748577+0000", 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012479599999999999 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749811+0000", 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012339530000000001 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20800 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14614528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723614+0000", 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.91147852, 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025236848999999999, 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.744 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20800 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723614+0000", 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723614+0000", 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723615+0000", 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.209e-06 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723620+0000", 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1989999999999997e-06 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723635+0000", 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723643+0000", 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1659999999999994e-06 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748682+0000", 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025039446 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748771+0000", 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.878e-05 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748850+0000", 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9561000000000006e-05 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20801 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14778368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723740+0000", 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9113521150000001, 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02527865, 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20801 2024-09-04T21:54:22.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723740+0000", 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723740+0000", 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723741+0000", 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3399999999999998e-07 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723744+0000", 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7079999999999999e-06 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723759+0000", 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4364e-05 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723767+0000", 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748879+0000", 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025112025999999999 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748965+0000", 2024-09-04T21:54:22.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5673000000000005e-05 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749019+0000", 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3523000000000002e-05 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20802 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14942208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9113125640000002, 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025380264999999999, 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20802 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723783+0000", 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8629999999999999e-06 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723791+0000", 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6550000000000004e-06 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723796+0000", 2024-09-04T21:54:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7520000000000002e-06 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749043+0000", 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025246951 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749099+0000", 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5179000000000001e-05 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749160+0000", 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1241999999999994e-05 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20803 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 15187968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723805+0000", 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9112871120000001, 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025468752000000001, 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20803 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723805+0000", 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723805+0000", 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723806+0000", 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4000000000000002e-07 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723808+0000", 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3429999999999998e-06 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723815+0000", 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9999999999999999e-06 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723867+0000", 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2061000000000002e-05 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749175+0000", 2024-09-04T21:54:22.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025308130000000002 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749218+0000", 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2945999999999998e-05 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749274+0000", 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5732000000000001e-05 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20804 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 15351808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723881+0000", 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9112109779999999, 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0255727, 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.763 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20804 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723881+0000", 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723881+0000", 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723882+0000", 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5400000000000001e-07 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723885+0000", 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4809999999999998e-06 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723894+0000", 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9900000000000003e-06 2024-09-04T21:54:22.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723901+0000", 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3330000000000002e-06 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749302+0000", 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025401182000000001 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749378+0000", 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6360999999999996e-05 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749454+0000", 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5698999999999995e-05 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20805 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:22.767 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.9111824209999999, 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025892301, 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20805 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723913+0000", 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8260000000000001e-06 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723923+0000", 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7159999999999998e-06 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723929+0000", 2024-09-04T21:54:22.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7370000000000003e-06 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749483+0000", 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025553868 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749569+0000", 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6494000000000002e-05 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:22.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749802+0000", 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023295999999999999 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:22.772 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:22.772 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:22.773 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-04T21:54:22.773 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-04T21:54:22.773 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-04T21:54:22.773 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-04T21:54:22.774 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:22.774+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:22.775 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:22.776 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone9 from 0xd6c4b46 (overlap 0x161954f) to 0xf7821d 2024-09-04T21:54:22.777 INFO:tasks.ceph.osd.7:Started 2024-09-04T21:54:22.777 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:22.875 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:22.874+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:22.878 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-09-04T21:54:22.897 INFO:teuthology.orchestra.run.smithi157.stderr:no valid command found; 10 closest matches: 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:0 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:1 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:2 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:abort 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:assert 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:config diff 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:config diff get 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:config get 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:config help [] 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:config set ... 2024-09-04T21:54:22.898 INFO:teuthology.orchestra.run.smithi157.stderr:admin_socket: invalid command 2024-09-04T21:54:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:54:22.900 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-04T21:54:22.906 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:22.906 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:22.906 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:22.906 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:22.920 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:22.933 INFO:teuthology.orchestra.run.smithi112.stdout:1861 read 0x2255a thru 0x2676f (0x4216 bytes) 2024-09-04T21:54:22.936 INFO:teuthology.orchestra.run.smithi112.stdout:1863 write 0x51aca5d thru 0x51b654c (0x9af0 bytes) 2024-09-04T21:54:22.945 INFO:teuthology.orchestra.run.smithi112.stdout:1864 read 0x9bda1bc thru 0x9be4721 (0xa566 bytes) 2024-09-04T21:54:22.945 INFO:teuthology.orchestra.run.smithi112.stdout:1865 write 0x4e06426 thru 0x4e0f4d0 (0x90ab bytes) 2024-09-04T21:54:22.949 INFO:teuthology.orchestra.run.smithi112.stdout:1866 read 0x4883e1 thru 0x4938db (0xb4fb bytes) 2024-09-04T21:54:22.949 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.950 INFO:teuthology.orchestra.run.smithi112.stdout:1870 trunc from 0xd6c4b46 to 0xdee2f72 2024-09-04T21:54:22.956 INFO:teuthology.orchestra.run.smithi112.stdout:1871 write 0xbc10f7e thru 0xbc1b429 (0xa4ac bytes) 2024-09-04T21:54:22.960 INFO:teuthology.orchestra.run.smithi112.stdout:1872 read 0x7e6b93f thru 0x7e790bd (0xd77f bytes) 2024-09-04T21:54:22.975 INFO:teuthology.orchestra.run.smithi112.stdout:1873 trunc from 0xdee2f72 to 0x3fa17ab 2024-09-04T21:54:22.975 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:22.971+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:22.983 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:22.984 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:1875 read 0x398e029 thru 0x399970c (0xb6e4 bytes) 2024-09-04T21:54:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:1877 punch from 0x1e7a873 to 0x1e871bf, (0xc94c bytes) 2024-09-04T21:54:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:1878 read 0x3ef2421 thru 0x3f00dba (0xe99a bytes) 2024-09-04T21:54:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:1879 punch from 0x18b1c9c to 0x18bbab6, (0x9e1a bytes) 2024-09-04T21:54:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:1880 read 0x19dedc6 thru 0x19ed2cc (0xe507 bytes) 2024-09-04T21:54:22.993 INFO:teuthology.orchestra.run.smithi112.stdout:1881 read 0x232265 thru 0x23f33f (0xd0db bytes) 2024-09-04T21:54:23.003 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:23.003 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:23.009 INFO:teuthology.orchestra.run.smithi112.stdout:1882 writesame 0xac43992 thru 0xac4f58e data_size 0x7d(0xbbfd bytes) 2024-09-04T21:54:23.009 INFO:teuthology.orchestra.run.smithi112.stdout:1883 read 0x5bb60ba thru 0x5bc2e7e (0xcdc5 bytes) 2024-09-04T21:54:23.011 INFO:teuthology.orchestra.run.smithi112.stdout:1884 trunc from 0xac4f58f to 0xc781a68 2024-09-04T21:54:23.017 INFO:teuthology.orchestra.run.smithi112.stdout:1885 read 0xb8d701c thru 0xb8dd842 (0x6827 bytes) 2024-09-04T21:54:23.017 INFO:teuthology.orchestra.run.smithi112.stdout:1887 read 0x8d7bbb4 thru 0x8d80b0b (0x4f58 bytes) 2024-09-04T21:54:23.019 INFO:teuthology.orchestra.run.smithi112.stdout:1888 trunc from 0xc781a68 to 0xd9e13af 2024-09-04T21:54:23.021 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:23.021 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:23.024 INFO:teuthology.orchestra.run.smithi112.stdout:1889 read 0x3088248 thru 0x309407d (0xbe36 bytes) 2024-09-04T21:54:23.025 INFO:teuthology.orchestra.run.smithi112.stdout:1890 read 0x3fd51e0 thru 0x3fde8a8 (0x96c9 bytes) 2024-09-04T21:54:23.025 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.026 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.026 INFO:teuthology.orchestra.run.smithi112.stdout:1893 read 0x1e1f900 thru 0x1e2b37f (0xba80 bytes) 2024-09-04T21:54:23.026 INFO:teuthology.orchestra.run.smithi112.stdout:1894 punch from 0x333de48 to 0x334d29b, (0xf453 bytes) 2024-09-04T21:54:23.026 INFO:teuthology.orchestra.run.smithi112.stdout:1895 punch from 0x10ee2a1 to 0x10f2839, (0x4598 bytes) 2024-09-04T21:54:23.026 INFO:teuthology.orchestra.run.smithi112.stdout:1896 write 0x43966b thru 0x444b80 (0xb516 bytes) 2024-09-04T21:54:23.032 INFO:teuthology.orchestra.run.smithi112.stdout:1897 punch from 0x299e31f to 0x29aaac0, (0xc7a1 bytes) 2024-09-04T21:54:23.040 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:23.040 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:23.041 INFO:teuthology.orchestra.run.smithi112.stdout:1899 trunc from 0xd9e13af to 0x73e5ea4 2024-09-04T21:54:23.047 INFO:teuthology.orchestra.run.smithi112.stdout:1900 read 0x39708be thru 0x39727b4 (0x1ef7 bytes) 2024-09-04T21:54:23.048 INFO:teuthology.orchestra.run.smithi112.stdout:1901 punch from 0x3acba9f to 0x3ada3ec, (0xe94d bytes) 2024-09-04T21:54:23.058 INFO:teuthology.orchestra.run.smithi112.stdout:1902 writesame 0x93c35ba thru 0x93cf66e data_size 0x1(0xc0b5 bytes) 2024-09-04T21:54:23.058 INFO:teuthology.orchestra.run.smithi112.stdout:1903 punch from 0x512c74b to 0x51325ef, (0x5ea4 bytes) 2024-09-04T21:54:23.058 INFO:teuthology.orchestra.run.smithi112.stdout:1904 punch from 0x2050802 to 0x205fa35, (0xf233 bytes) 2024-09-04T21:54:23.058 INFO:teuthology.orchestra.run.smithi112.stdout:1905 punch from 0x2e85d35 to 0x2e930fb, (0xd3c6 bytes) 2024-09-04T21:54:23.062 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:23.063 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:23.067 INFO:teuthology.orchestra.run.smithi112.stdout:1906 trunc from 0x93cf66f to 0x2d8fb63 2024-09-04T21:54:23.076 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:23.075+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:23.082 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:23.082 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:23.090 INFO:teuthology.orchestra.run.smithi112.stdout:1908 read 0x5e305a thru 0x5ef59a (0xc541 bytes) 2024-09-04T21:54:23.091 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:23.091 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:23.100 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:23.101 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:23.101 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:23.102 INFO:teuthology.orchestra.run.smithi112.stdout:1911 write 0x5f4afb3 thru 0x5f55655 (0xa6a3 bytes) 2024-09-04T21:54:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:23.105 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:23.106 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:1913 punch from 0x49cec45 to 0x49d81d8, (0x9593 bytes) 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:1914 read 0x3a47db4 thru 0x3a4ced3 (0x5120 bytes) 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:1915 read 0x52035a7 thru 0x5210fc1 (0xda1b bytes) 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:1918 punch from 0x3e94635 to 0x3e9d2dd, (0x8ca8 bytes) 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.108 INFO:teuthology.orchestra.run.smithi112.stdout:1920 read 0x8b4a97 thru 0x8b9173 (0x46dd bytes) 2024-09-04T21:54:23.109 INFO:teuthology.orchestra.run.smithi112.stdout:1925 read 0x312a577 thru 0x312bc68 (0x16f2 bytes) 2024-09-04T21:54:23.114 INFO:teuthology.orchestra.run.smithi112.stdout:1928 trunc from 0x5f55656 to 0x1d5f702 2024-09-04T21:54:23.135 INFO:teuthology.orchestra.run.smithi112.stdout:1929 trunc from 0x1d5f702 to 0x3c22c36 2024-09-04T21:54:23.142 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.160 INFO:teuthology.orchestra.run.smithi112.stdout:1931 write 0xb4156ab thru 0xb422934 (0xd28a bytes) 2024-09-04T21:54:23.164 INFO:teuthology.orchestra.run.smithi112.stdout:1932 punch from 0xa5b04f8 to 0xa5b08b4, (0x3bc bytes) 2024-09-04T21:54:23.165 INFO:teuthology.orchestra.run.smithi112.stdout:1933 write 0x85f438f thru 0x8602b38 (0xe7aa bytes) 2024-09-04T21:54:23.177 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:23.174+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:23.199 INFO:teuthology.orchestra.run.smithi112.stdout:1934 write 0xc7766e7 thru 0xc77e80c (0x8126 bytes) 2024-09-04T21:54:23.199 INFO:teuthology.orchestra.run.smithi112.stdout:1935 write 0xdc9e1c5 thru 0xdca7c86 (0x9ac2 bytes) 2024-09-04T21:54:23.200 INFO:teuthology.orchestra.run.smithi112.stdout:1936 read 0x8ccbe34 thru 0x8ccbf7e (0x14b bytes) 2024-09-04T21:54:23.200 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.200 INFO:teuthology.orchestra.run.smithi112.stdout:1938 writesame 0x7dff038 thru 0x7e08531 data_size 0x1(0x94fa bytes) 2024-09-04T21:54:23.224 INFO:teuthology.orchestra.run.smithi112.stdout:1939 write 0xdf235e4 thru 0xdf3245b (0xee78 bytes) 2024-09-04T21:54:23.229 INFO:teuthology.orchestra.run.smithi112.stdout:1940 write 0xb650907 thru 0xb65df4c (0xd646 bytes) 2024-09-04T21:54:23.232 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:23.232 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:23.234 INFO:teuthology.orchestra.run.smithi112.stdout:1941 read 0x6727a83 thru 0x67286af (0xc2d bytes) 2024-09-04T21:54:23.234 INFO:teuthology.orchestra.run.smithi112.stdout:1942 read 0xd0e3947 thru 0xd0f22b3 (0xe96d bytes) 2024-09-04T21:54:23.234 INFO:teuthology.orchestra.run.smithi112.stdout:1943 read 0x275f98d thru 0x276a126 (0xa79a bytes) 2024-09-04T21:54:23.246 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:23.265 INFO:teuthology.orchestra.run.smithi112.stdout:1945 trunc from 0xdf3245c to 0x1e621be 2024-09-04T21:54:23.294 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:23.294+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:23.312 INFO:teuthology.orchestra.run.smithi112.stdout:1946 write 0x9a038ee thru 0x9a04b58 (0x126b bytes) 2024-09-04T21:54:23.318 INFO:teuthology.orchestra.run.smithi112.stdout:1948 trunc from 0x9a04b59 to 0x8e5a49f 2024-09-04T21:54:23.356 INFO:teuthology.orchestra.run.smithi112.stdout:1952 write 0xc4c6d8b thru 0xc4d20f4 (0xb36a bytes) 2024-09-04T21:54:23.365 INFO:teuthology.orchestra.run.smithi112.stdout:1953 read 0x978751f thru 0x9795fc3 (0xeaa5 bytes) 2024-09-04T21:54:23.365 INFO:teuthology.orchestra.run.smithi112.stdout:1954 write 0x79a4c91 thru 0x79ae0b1 (0x9421 bytes) 2024-09-04T21:54:23.373 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:23.373 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:23.373 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.379 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:23.378+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:23.388 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:23.393 INFO:teuthology.orchestra.run.smithi112.stdout:1957 write 0xe6c566c thru 0xe6cd6b0 (0x8045 bytes) 2024-09-04T21:54:23.431 INFO:teuthology.orchestra.run.smithi112.stdout:1958 trunc from 0xe6cd6b1 to 0x2cf0cf2 2024-09-04T21:54:23.480 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:23.478+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:23.484 INFO:teuthology.orchestra.run.smithi112.stdout:1959 writesame 0xb13b2ce thru 0xb13e7c3 data_size 0x1(0x34f6 bytes) 2024-09-04T21:54:23.484 INFO:teuthology.orchestra.run.smithi112.stdout:1960 punch from 0x6674a38 to 0x6683ec4, (0xf48c bytes) 2024-09-04T21:54:23.486 INFO:teuthology.orchestra.run.smithi112.stdout:1961 trunc from 0xb13e7c4 to 0xa5625a4 2024-09-04T21:54:23.508 INFO:teuthology.orchestra.run.smithi112.stdout:1962 writesame 0xa62f96d thru 0xa63edf0 data_size 0x2(0xf484 bytes) 2024-09-04T21:54:23.516 INFO:teuthology.orchestra.run.smithi112.stdout:1963 trunc from 0xa63edf1 to 0x74fd3fd 2024-09-04T21:54:23.530 INFO:teuthology.orchestra.run.smithi112.stdout:1964 read 0x47ae16c thru 0x47b2fd0 (0x4e65 bytes) 2024-09-04T21:54:23.530 INFO:teuthology.orchestra.run.smithi112.stdout:1965 punch from 0x5e2a376 to 0x5e2f0ce, (0x4d58 bytes) 2024-09-04T21:54:23.530 INFO:teuthology.orchestra.run.smithi112.stdout:1966 read 0x29c3aa0 thru 0x29cf0b8 (0xb619 bytes) 2024-09-04T21:54:23.530 INFO:teuthology.orchestra.run.smithi112.stdout:1967 read 0x5fc63fd thru 0x5fd280c (0xc410 bytes) 2024-09-04T21:54:23.530 INFO:teuthology.orchestra.run.smithi112.stdout:1968 punch from 0x4c7ed24 to 0x4c8b9e2, (0xccbe bytes) 2024-09-04T21:54:23.543 INFO:teuthology.orchestra.run.smithi112.stdout:1969 trunc from 0x74fd3fd to 0xc39e7f6 2024-09-04T21:54:23.553 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:23.553 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:23.554 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.554 INFO:teuthology.orchestra.run.smithi112.stdout:1974 writesame 0x87914d9 thru 0x87955f6 data_size 0x5(0x411e bytes) 2024-09-04T21:54:23.562 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:23.562 INFO:teuthology.orchestra.run.smithi112.stdout:1978 read 0xf16dd2 thru 0xf24197 (0xd3c6 bytes) 2024-09-04T21:54:23.564 INFO:teuthology.orchestra.run.smithi112.stdout:1980 write 0x7114d85 thru 0x7122f2a (0xe1a6 bytes) 2024-09-04T21:54:23.569 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:23.573 INFO:teuthology.orchestra.run.smithi112.stdout:1983 punch from 0x2e5ab2c to 0x2e5b7f1, (0xcc5 bytes) 2024-09-04T21:54:23.573 INFO:teuthology.orchestra.run.smithi112.stdout:1987 read 0xa22a8c1 thru 0xa22b795 (0xed5 bytes) 2024-09-04T21:54:23.587 INFO:teuthology.orchestra.run.smithi112.stdout:1989 write 0xd7a9bf1 thru 0xd7ae014 (0x4424 bytes) 2024-09-04T21:54:23.592 INFO:teuthology.orchestra.run.smithi112.stdout:1990 write 0x868fe3b thru 0x86977c3 (0x7989 bytes) 2024-09-04T21:54:23.609 INFO:teuthology.orchestra.run.smithi112.stdout:1991 trunc from 0xd7ae015 to 0x89fce69 2024-09-04T21:54:23.615 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:23.614+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:23.628 INFO:teuthology.orchestra.run.smithi112.stdout:1992 read 0x7277cdc thru 0x72813ae (0x96d3 bytes) 2024-09-04T21:54:23.641 INFO:teuthology.orchestra.run.smithi112.stdout:1993 trunc from 0x89fce69 to 0x3ee7369 2024-09-04T21:54:23.682 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:23.678+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:23.693 INFO:teuthology.orchestra.run.smithi112.stdout:1995 write 0xa9614d0 thru 0xa970367 (0xee98 bytes) 2024-09-04T21:54:23.702 INFO:teuthology.orchestra.run.smithi112.stdout:1996 read 0x84a6443 thru 0x84aebc2 (0x8780 bytes) 2024-09-04T21:54:23.711 INFO:teuthology.orchestra.run.smithi112.stdout:1997 trunc from 0xa970368 to 0xd850eef 2024-09-04T21:54:23.724 INFO:teuthology.orchestra.run.smithi112.stdout:1999 read 0x3653850 thru 0x3653d8c (0x53d bytes) 2024-09-04T21:54:23.724 INFO:teuthology.orchestra.run.smithi112.stdout:2000 read 0xa855173 thru 0xa8593c8 (0x4256 bytes) 2024-09-04T21:54:23.739 INFO:teuthology.orchestra.run.smithi112.stdout:2001 trunc from 0xd850eef to 0x812f2b1 2024-09-04T21:54:23.744 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:23.744 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:23.761 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:2002 read 0x80272c thru 0x80a9b6 (0x828b bytes) 2024-09-04T21:54:23.763 INFO:teuthology.orchestra.run.smithi112.stdout:2003 punch from 0x58e07b4 to 0x58e79aa, (0x71f6 bytes) 2024-09-04T21:54:23.783 INFO:teuthology.orchestra.run.smithi112.stdout:2004 trunc from 0x812f2b1 to 0xa218bd 2024-09-04T21:54:23.806 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:23.806+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:23.813 INFO:teuthology.orchestra.run.smithi112.stdout:2005 punch from 0x5d60ca to 0x5d744d, (0x1383 bytes) 2024-09-04T21:54:23.856 INFO:teuthology.orchestra.run.smithi112.stdout:2006 writesame 0xd4f60bc thru 0xd4ffc64 data_size 0x25(0x9ba9 bytes) 2024-09-04T21:54:23.857 INFO:teuthology.orchestra.run.smithi112.stdout:2008 write 0x928da1 thru 0x92b6f7 (0x2957 bytes) 2024-09-04T21:54:23.873 INFO:teuthology.orchestra.run.smithi112.stdout:2009 trunc from 0xd4ffc65 to 0xab06d0d 2024-09-04T21:54:23.883 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:23.882+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:23.889 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:23.889 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:23.890 INFO:teuthology.orchestra.run.smithi112.stdout:2010 writesame 0x1b24788 thru 0x1b2de02 data_size 0x1(0x967b bytes) 2024-09-04T21:54:23.904 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:23.927 INFO:teuthology.orchestra.run.smithi112.stdout:2011 read 0x7d03a5d thru 0x7d0d785 (0x9d29 bytes) 2024-09-04T21:54:23.928 INFO:teuthology.orchestra.run.smithi112.stdout:2012 punch from 0x9dec2cc to 0x9dec4de, (0x212 bytes) 2024-09-04T21:54:23.928 INFO:teuthology.orchestra.run.smithi112.stdout:2013 clone 11 order 25 su 262144 sc 7 2024-09-04T21:54:23.994 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:23.994+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:24.068 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:24.069 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:24.082 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:24.126 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:24.126+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:24.187 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:24.186+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:24.244 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:24.244 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:24.258 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:24.304 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:24.302+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:24.389 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:24.386+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:24.390 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:24.390 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:24.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:24.491 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:24.490+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:24.546 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:54:24.579 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:24.591 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:24.586+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:24.596 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:24.680 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone10 from 0xab06d0d (overlap 0xa218bd) to 0x81ad64 2024-09-04T21:54:24.685 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:24.682+0000 7fb39a3ba7c0 -1 Falling back to public interface 2024-09-04T21:54:24.691 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:24.686+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:24.779 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:24.779 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:24.793 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:24.790+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:24.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:24.893 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:24.890+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:24.907 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-09-04T21:54:24.951 INFO:teuthology.orchestra.run.smithi112.stdout:2014 read 0x3550df1 thru 0x355a36e (0x957e bytes) 2024-09-04T21:54:24.952 INFO:teuthology.orchestra.run.smithi112.stdout:2015 punch from 0x3fbb6b to 0x405a1d, (0x9eb2 bytes) 2024-09-04T21:54:24.960 INFO:teuthology.orchestra.run.smithi112.stdout:2016 read 0xa70dbfa thru 0xa71081f (0x2c26 bytes) 2024-09-04T21:54:24.971 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:24.971 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:24.973 INFO:teuthology.orchestra.run.smithi112.stdout:2017 writesame 0xb0240a9 thru 0xb03099d data_size 0x1(0xc8f5 bytes) 2024-09-04T21:54:24.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout:2018 write 0x9aea6c1 thru 0x9af1b41 (0x7481 bytes) 2024-09-04T21:54:24.996 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:24.990+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:25.032 INFO:teuthology.orchestra.run.smithi112.stdout:2019 trunc from 0xb03099e to 0x198c3ac 2024-09-04T21:54:25.094 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:25.094+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:25.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:25.113 INFO:teuthology.orchestra.run.smithi112.stdout:2021 write 0x94e4796 thru 0x94eaac4 (0x632f bytes) 2024-09-04T21:54:25.116 INFO:teuthology.orchestra.run.smithi112.stdout:2023 write 0x911a7e0 thru 0x912762f (0xce50 bytes) 2024-09-04T21:54:25.119 INFO:teuthology.orchestra.run.smithi112.stdout:2024 punch from 0x8a52cdf to 0x8a55207, (0x2528 bytes) 2024-09-04T21:54:25.119 INFO:teuthology.orchestra.run.smithi112.stdout:2025 punch from 0x8de053a to 0x8de7cb9, (0x777f bytes) 2024-09-04T21:54:25.120 INFO:teuthology.orchestra.run.smithi112.stdout:2026 flatten 2024-09-04T21:54:25.133 INFO:teuthology.orchestra.run.smithi112.stdout:2027 writesame 0x79b5dbc thru 0x79c4d2f data_size 0x32(0xef74 bytes) 2024-09-04T21:54:25.137 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.137 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.137 INFO:teuthology.orchestra.run.smithi112.stdout:2031 write 0x3714b45 thru 0x371b571 (0x6a2d bytes) 2024-09-04T21:54:25.140 INFO:teuthology.orchestra.run.smithi112.stdout:2034 write 0x201b21e thru 0x201f9e8 (0x47cb bytes) 2024-09-04T21:54:25.141 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:25.141 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:25.142 INFO:teuthology.orchestra.run.smithi112.stdout:2035 read 0x3bef323 thru 0x3bf93e3 (0xa0c1 bytes) 2024-09-04T21:54:25.143 INFO:teuthology.orchestra.run.smithi112.stdout:2036 punch from 0x6f568a to 0x703af0, (0xe466 bytes) 2024-09-04T21:54:25.143 INFO:teuthology.orchestra.run.smithi112.stdout:2038 read 0x165dfd0 thru 0x166a548 (0xc579 bytes) 2024-09-04T21:54:25.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:25.162 INFO:teuthology.orchestra.run.smithi112.stdout:2039 write 0xbf20fd9 thru 0xbf2e00c (0xd034 bytes) 2024-09-04T21:54:25.164 INFO:teuthology.orchestra.run.smithi112.stdout:2041 read 0xbe3499e thru 0xbe3923d (0x48a0 bytes) 2024-09-04T21:54:25.165 INFO:teuthology.orchestra.run.smithi112.stdout:2042 write 0x490d146 thru 0x490d4c9 (0x384 bytes) 2024-09-04T21:54:25.167 INFO:teuthology.orchestra.run.smithi112.stdout:2043 writesame 0x9189ad3 thru 0x918b15c data_size 0x5(0x168a bytes) 2024-09-04T21:54:25.170 INFO:teuthology.orchestra.run.smithi112.stdout:2044 write 0x28e4a0 thru 0x296ae0 (0x8641 bytes) 2024-09-04T21:54:25.188 INFO:teuthology.orchestra.run.smithi112.stdout:2045 writesame 0xdb0b2ab thru 0xdb0d2d5 data_size 0x1b(0x202b bytes) 2024-09-04T21:54:25.190 INFO:teuthology.orchestra.run.smithi112.stdout:2046 read 0x40285d2 thru 0x40309a0 (0x83cf bytes) 2024-09-04T21:54:25.191 INFO:teuthology.orchestra.run.smithi112.stdout:2048 punch from 0x8186d18 to 0x8191438, (0xa720 bytes) 2024-09-04T21:54:25.191 INFO:teuthology.orchestra.run.smithi112.stdout:2050 read 0xc0329f8 thru 0xc032a54 (0x5d bytes) 2024-09-04T21:54:25.192 INFO:teuthology.orchestra.run.smithi112.stdout:2051 punch from 0xd680abe to 0xd68c912, (0xbe54 bytes) 2024-09-04T21:54:25.192 INFO:teuthology.orchestra.run.smithi112.stdout:2052 read 0x3463b30 thru 0x346e15d (0xa62e bytes) 2024-09-04T21:54:25.195 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:25.194+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:25.198 INFO:teuthology.orchestra.run.smithi112.stdout:2053 trunc from 0xdb0d2d6 to 0x9ffc17a 2024-09-04T21:54:25.212 INFO:teuthology.orchestra.run.smithi112.stdout:2054 read 0x6ab9768 thru 0x6abdd3d (0x45d6 bytes) 2024-09-04T21:54:25.215 INFO:teuthology.orchestra.run.smithi112.stdout:2056 trunc from 0x9ffc17a to 0xb8be545 2024-09-04T21:54:25.240 INFO:teuthology.orchestra.run.smithi112.stdout:2058 write 0xd422f6d thru 0xd42e1c5 (0xb259 bytes) 2024-09-04T21:54:25.243 INFO:teuthology.orchestra.run.smithi112.stdout:2060 write 0x8c45ac9 thru 0x8c54579 (0xeab1 bytes) 2024-09-04T21:54:25.246 INFO:teuthology.orchestra.run.smithi112.stdout:2063 read 0x9a2e804 thru 0x9a3a752 (0xbf4f bytes) 2024-09-04T21:54:25.246 INFO:teuthology.orchestra.run.smithi112.stdout:2065 write 0x75db252 thru 0x75deb77 (0x3926 bytes) 2024-09-04T21:54:25.250 INFO:teuthology.orchestra.run.smithi112.stdout:2066 writesame 0x8def139 thru 0x8df1fe6 data_size 0x19(0x2eae bytes) 2024-09-04T21:54:25.252 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.252 INFO:teuthology.orchestra.run.smithi112.stdout:2069 write 0x8671c2 thru 0x873162 (0xbfa1 bytes) 2024-09-04T21:54:25.254 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:25.255 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:25.255 INFO:teuthology.orchestra.run.smithi112.stdout:2072 read 0x20f0da6 thru 0x20f5e36 (0x5091 bytes) 2024-09-04T21:54:25.256 INFO:teuthology.orchestra.run.smithi112.stdout:2073 read 0xbe71f89 thru 0xbe7af5f (0x8fd7 bytes) 2024-09-04T21:54:25.257 INFO:teuthology.orchestra.run.smithi112.stdout:2074 read 0x99f6d thru 0xa05ad (0x6641 bytes) 2024-09-04T21:54:25.258 INFO:teuthology.orchestra.run.smithi112.stdout:2076 punch from 0x1f6f182 to 0x1f7ec33, (0xfab1 bytes) 2024-09-04T21:54:25.258 INFO:teuthology.orchestra.run.smithi112.stdout:2077 punch from 0xd301581 to 0xd30a449, (0x8ec8 bytes) 2024-09-04T21:54:25.258 INFO:teuthology.orchestra.run.smithi112.stdout:2078 punch from 0x8623005 to 0x8627206, (0x4201 bytes) 2024-09-04T21:54:25.258 INFO:teuthology.orchestra.run.smithi112.stdout:2079 write 0xb81a003 thru 0xb81f2c0 (0x52be bytes) 2024-09-04T21:54:25.261 INFO:teuthology.orchestra.run.smithi112.stdout:2080 writesame 0x5a13d9a thru 0x5a1a123 data_size 0x5d(0x638a bytes) 2024-09-04T21:54:25.262 INFO:teuthology.orchestra.run.smithi112.stdout:2081 write 0xb19da7b thru 0xb1a4e75 (0x73fb bytes) 2024-09-04T21:54:25.264 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.264 INFO:teuthology.orchestra.run.smithi112.stdout:2083 write 0xa1f0d1a thru 0xa1fab45 (0x9e2c bytes) 2024-09-04T21:54:25.265 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.265 INFO:teuthology.orchestra.run.smithi112.stdout:2086 read 0x5290468 thru 0x5297efa (0x7a93 bytes) 2024-09-04T21:54:25.266 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.266 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.266 INFO:teuthology.orchestra.run.smithi112.stdout:2090 read 0x2e41187 thru 0x2e4f7b0 (0xe62a bytes) 2024-09-04T21:54:25.274 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:25.274 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:25.281 INFO:teuthology.orchestra.run.smithi112.stdout:2091 trunc from 0xd42e1c6 to 0x3359950 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.438284991, 2024-09-04T21:54:25.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:25.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.312237727999999, 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.299 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:25.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:25.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.115684089999998, 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.305 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:25.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.088294592, 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.065030120999999, 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:25.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.037272331, 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:25.322 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:25.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:25.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.329 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.037156768999999, 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:25.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 21.736981984, 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.577849432000001, 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.342 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:25.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:25.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.577623374000002, 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:25.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:25.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.100410306000001, 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.354 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.871022194999998, 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:25.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.572831554, 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.367 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:25.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:25.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.572718698999999, 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:25.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.563057731000001, 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.379 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:25.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.543208006, 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:25.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.392979319999998, 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:25.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:25.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.393 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.726652415, 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:25.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:25.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:25.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:25.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.723513229000002, 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.705202238999998, 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.404 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:25.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.409 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:25.409 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:25.410 INFO:teuthology.orchestra.run.smithi112.stdout:2093 trunc from 0x3359950 to 0x588693a 2024-09-04T21:54:25.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:25.411 INFO:teuthology.orchestra.run.smithi112.stdout:2094 read 0x12b062 thru 0x130df9 (0x5d98 bytes) 2024-09-04T21:54:25.412 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:25.412 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2095 writesame 0xdfa86e1 thru 0xdfb0f5a data_size 0x1b(0x887a bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2096 read 0x414c948 thru 0x415c714 (0xfdcd bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2097 read 0x9e4758 thru 0x9e7cbf (0x3568 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2098 write 0x5b6fd0 thru 0x5c2fa0 (0xbfd1 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2099 read 0x7ed40a7 thru 0x7eda288 (0x61e2 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2100 write 0x207b2d thru 0x207c70 (0x144 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2103 read 0x17ca350 thru 0x17cd187 (0x2e38 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2105 read 0x1b7f452 thru 0x1b87393 (0x7f42 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2106 write 0x85fd164 thru 0x85fe35c (0x11f9 bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2107 write 0xbaff97c thru 0xbb0f197 (0xf81c bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2108 writesame 0x32e5b95 thru 0x32f38fe data_size 0x12(0xdd6a bytes) 2024-09-04T21:54:25.414 INFO:teuthology.orchestra.run.smithi112.stdout:2111 write 0xa8912aa thru 0xa892024 (0xd7b bytes) 2024-09-04T21:54:25.415 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:2112 read 0x6880b1e thru 0x688eb66 (0xe049 bytes) 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:2114 write 0xffcea7 thru 0xffd133 (0x28d bytes) 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:2116 read 0x55b57e1 thru 0x55b900f (0x382f bytes) 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:2118 write 0xd334cb0 thru 0xd340a50 (0xbda1 bytes) 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:2121 trunc from 0xdfb0f5b to 0xc1a9f22 2024-09-04T21:54:25.416 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:25.417 INFO:teuthology.orchestra.run.smithi112.stdout:2122 writesame 0x39ecfd thru 0x3a86fd data_size 0x19(0x9a01 bytes) 2024-09-04T21:54:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:25.418 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:25.418 INFO:teuthology.orchestra.run.smithi112.stdout:2123 writesame 0x1249dd2 thru 0x124ba12 data_size 0x1(0x1c41 bytes) 2024-09-04T21:54:25.418 INFO:teuthology.orchestra.run.smithi112.stdout:2124 trunc from 0xc1a9f22 to 0x85d8e5d 2024-09-04T21:54:25.419 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:25.406+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:25.419 INFO:teuthology.orchestra.run.smithi112.stdout:2128 punch from 0x833c890 to 0x8344170, (0x78e0 bytes) 2024-09-04T21:54:25.426 INFO:teuthology.orchestra.run.smithi112.stdout:2129 trunc from 0x85d8e5d to 0xd9ec63b 2024-09-04T21:54:25.437 INFO:teuthology.orchestra.run.smithi112.stdout:2130 write 0x65c40de thru 0x65cafde (0x6f01 bytes) 2024-09-04T21:54:25.440 INFO:teuthology.orchestra.run.smithi112.stdout:2131 writesame 0x3595605 thru 0x35a0d32 data_size 0x1(0xb72e bytes) 2024-09-04T21:54:25.498 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.499 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.499 INFO:teuthology.orchestra.run.smithi112.stdout:2136 read 0x44cc058 thru 0x44ce34b (0x22f4 bytes) 2024-09-04T21:54:25.499 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.499 INFO:teuthology.orchestra.run.smithi112.stdout:2139 read 0x7b9199d thru 0x7b9cc3d (0xb2a1 bytes) 2024-09-04T21:54:25.511 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:25.510+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:25.513 INFO:teuthology.orchestra.run.smithi112.stdout:2140 trunc from 0xd9ec63b to 0x51c3a08 2024-09-04T21:54:25.529 INFO:teuthology.orchestra.run.smithi112.stdout:2141 read 0x502098e thru 0x50245f8 (0x3c6b bytes) 2024-09-04T21:54:25.530 INFO:teuthology.orchestra.run.smithi112.stdout:2142 read 0x3581c6a thru 0x35832b6 (0x164d bytes) 2024-09-04T21:54:25.531 INFO:teuthology.orchestra.run.smithi112.stdout:2144 read 0xf8c30a thru 0xf90e00 (0x4af7 bytes) 2024-09-04T21:54:25.532 INFO:teuthology.orchestra.run.smithi112.stdout:2145 punch from 0x3f10676 to 0x3f1ec24, (0xe5ae bytes) 2024-09-04T21:54:25.532 INFO:teuthology.orchestra.run.smithi112.stdout:2146 read 0x4ebec1d thru 0x4ec7128 (0x850c bytes) 2024-09-04T21:54:25.544 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:54:25.546 INFO:teuthology.orchestra.run.smithi112.stdout:2149 trunc from 0x51c3a08 to 0xbf27d19 2024-09-04T21:54:25.558 INFO:teuthology.orchestra.run.smithi112.stdout:2153 writesame 0x8fd1444 thru 0x8fddaa8 data_size 0x1(0xc665 bytes) 2024-09-04T21:54:25.561 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:25.561 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:25.561 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:25.562 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:25.578 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:25.610 INFO:teuthology.orchestra.run.smithi112.stdout:2154 read 0xd51430 thru 0xd560b9 (0x4c8a bytes) 2024-09-04T21:54:25.612 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.612 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:25.606+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:25.622 INFO:teuthology.orchestra.run.smithi112.stdout:2156 write 0xcc7a234 thru 0xcc86ebf (0xcc8c bytes) 2024-09-04T21:54:25.625 INFO:teuthology.orchestra.run.smithi112.stdout:2157 writesame 0x20943d2 thru 0x20a201a data_size 0x1(0xdc49 bytes) 2024-09-04T21:54:25.681 INFO:teuthology.orchestra.run.smithi112.stdout:2158 read 0xace2681 thru 0xace863d (0x5fbd bytes) 2024-09-04T21:54:25.682 INFO:teuthology.orchestra.run.smithi112.stdout:2159 write 0xc305246 thru 0xc311616 (0xc3d1 bytes) 2024-09-04T21:54:25.686 INFO:teuthology.orchestra.run.smithi112.stdout:2160 write 0x6bd3d42 thru 0x6bd4016 (0x2d5 bytes) 2024-09-04T21:54:25.694 INFO:teuthology.orchestra.run.smithi112.stdout:2161 trunc from 0xcc86ec0 to 0x9d9bcb3 2024-09-04T21:54:25.706 INFO:teuthology.orchestra.run.smithi112.stdout:2162 read 0x1f3249 thru 0x1fab11 (0x78c9 bytes) 2024-09-04T21:54:25.713 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:25.710+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:25.720 INFO:teuthology.orchestra.run.smithi112.stdout:2163 trunc from 0x9d9bcb3 to 0x1e3c70a 2024-09-04T21:54:25.735 INFO:teuthology.orchestra.run.smithi112.stdout:2164 read 0x1c8a18e thru 0x1c98a46 (0xe8b9 bytes) 2024-09-04T21:54:25.736 INFO:teuthology.orchestra.run.smithi112.stdout:2167 read 0x14511e9 thru 0x1460963 (0xf77b bytes) 2024-09-04T21:54:25.737 INFO:teuthology.orchestra.run.smithi112.stdout:2168 read 0x180b513 thru 0x18125f5 (0x70e3 bytes) 2024-09-04T21:54:25.739 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:25.740 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:25.771 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:25.774 INFO:teuthology.orchestra.run.smithi112.stdout:2171 write 0xe5a4237 thru 0xe5b410b (0xfed5 bytes) 2024-09-04T21:54:25.779 INFO:teuthology.orchestra.run.smithi112.stdout:2172 write 0x7ae99d7 thru 0x7af4eeb (0xb515 bytes) 2024-09-04T21:54:25.782 INFO:teuthology.orchestra.run.smithi112.stdout:2174 read 0xa69d335 thru 0xa6a05c5 (0x3291 bytes) 2024-09-04T21:54:25.783 INFO:teuthology.orchestra.run.smithi112.stdout:2175 write 0x3089f03 thru 0x308a419 (0x517 bytes) 2024-09-04T21:54:25.785 INFO:teuthology.orchestra.run.smithi112.stdout:2176 read 0x5f5efe3 thru 0x5f64a4c (0x5a6a bytes) 2024-09-04T21:54:25.786 INFO:teuthology.orchestra.run.smithi112.stdout:2178 writesame 0x54afdfd thru 0x54b9913 data_size 0x1(0x9b17 bytes) 2024-09-04T21:54:25.809 INFO:teuthology.orchestra.run.smithi112.stdout:2179 write 0xa556ae0 thru 0xa55e926 (0x7e47 bytes) 2024-09-04T21:54:25.812 INFO:teuthology.orchestra.run.smithi112.stdout:2180 punch from 0xc79e5db to 0xc79fc1b, (0x1640 bytes) 2024-09-04T21:54:25.812 INFO:teuthology.orchestra.run.smithi112.stdout:2181 write 0x595705a thru 0x5965e27 (0xedce bytes) 2024-09-04T21:54:25.814 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:25.810+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:25.815 INFO:teuthology.orchestra.run.smithi112.stdout:2183 write 0xe4ba5c thru 0xe4f47a (0x3a1f bytes) 2024-09-04T21:54:25.817 INFO:teuthology.orchestra.run.smithi112.stdout:2185 write 0xb98b868 thru 0xb996e7d (0xb616 bytes) 2024-09-04T21:54:25.824 INFO:teuthology.orchestra.run.smithi112.stdout:2186 trunc from 0xe5b410c to 0xbdde45f 2024-09-04T21:54:25.845 INFO:teuthology.orchestra.run.smithi112.stdout:2187 punch from 0xb86a2e8 to 0xb876138, (0xbe50 bytes) 2024-09-04T21:54:25.845 INFO:teuthology.orchestra.run.smithi112.stdout:2188 write 0x2bd9a9e thru 0x2bd9b21 (0x84 bytes) 2024-09-04T21:54:25.848 INFO:teuthology.orchestra.run.smithi112.stdout:2189 punch from 0x6c2c0cc to 0x6c2c5af, (0x4e3 bytes) 2024-09-04T21:54:25.848 INFO:teuthology.orchestra.run.smithi112.stdout:2190 punch from 0x74e56e to 0x75d563, (0xeff5 bytes) 2024-09-04T21:54:25.848 INFO:teuthology.orchestra.run.smithi112.stdout:2191 read 0x97c4861 thru 0x97d2cd4 (0xe474 bytes) 2024-09-04T21:54:25.849 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:25.849 INFO:teuthology.orchestra.run.smithi112.stdout:2194 read 0x3a1434c thru 0x3a1c98a (0x863f bytes) 2024-09-04T21:54:25.850 INFO:teuthology.orchestra.run.smithi112.stdout:2195 write 0x14a6d18 thru 0x14b5ca0 (0xef89 bytes) 2024-09-04T21:54:25.852 INFO:teuthology.orchestra.run.smithi112.stdout:2197 write 0x1802ba5 thru 0x180b430 (0x888c bytes) 2024-09-04T21:54:25.856 INFO:teuthology.orchestra.run.smithi112.stdout:2199 write 0x2173002 thru 0x2175c7d (0x2c7c bytes) 2024-09-04T21:54:25.874 INFO:teuthology.orchestra.run.smithi112.stdout:2200 write 0xe4dc3b9 thru 0xe4e984e (0xd496 bytes) 2024-09-04T21:54:25.880 INFO:teuthology.orchestra.run.smithi112.stdout:2201 read 0xdd1f49b thru 0xdd25618 (0x617e bytes) 2024-09-04T21:54:25.881 INFO:teuthology.orchestra.run.smithi112.stdout:2202 punch from 0xd635e08 to 0xd63601a, (0x212 bytes) 2024-09-04T21:54:25.900 INFO:teuthology.orchestra.run.smithi112.stdout:2205 trunc from 0xe4e984f to 0x1bb0303 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7156 3.4 e30/22 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head v 30'66, mlcod=30'64)", 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.191534125, 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002957763, 2024-09-04T21:54:25.905 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713045+0000", 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713047+0000", 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.528e-06 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713248+0000", 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201287 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713277+0000", 2024-09-04T21:54:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8487e-05 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713288+0000", 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0832000000000001e-05 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713359+0000", 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0993000000000003e-05 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.713382+0000", 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3580000000000001e-05 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715959+0000", 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025766869999999998 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716003+0000", 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4369000000000002e-05 2024-09-04T21:54:25.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7159 3.9 e30/22 3:93c2ae1d:::rbd_data.118867f9efa3.000000000000000f:head v 30'14, mlcod=29'10)", 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.189364511000001, 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031393580000000001, 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715215+0000", 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715216+0000", 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6300000000000004e-07 2024-09-04T21:54:25.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715223+0000", 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2980000000000003e-06 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715237+0000", 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3361e-05 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715244+0000", 2024-09-04T21:54:25.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.362e-06 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716436+0000", 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00119187 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716456+0000", 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0562999999999999e-05 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718316+0000", 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018595720000000001 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718354+0000", 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8469e-05 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.916 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7729 3.4 e31/22 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head v 31'74, mlcod=30'72)", 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.006019390999999, 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031016809999999998, 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898560+0000", 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898561+0000", 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898620+0000", 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8813000000000001e-05 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898654+0000", 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4292999999999998e-05 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898675+0000", 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1024000000000001e-05 2024-09-04T21:54:25.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898834+0000", 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015876599999999999 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898899+0000", 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5209000000000005e-05 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901577+0000", 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026777849999999998 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901662+0000", 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4983999999999998e-05 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.921 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7863 3.9 e32/22 3:9478b05c:::rbd_data.1188b1294ada.000000000000005b:head v 32'21, mlcod=31'18)", 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.339948701000001, 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002738262, 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564631+0000", 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564632+0000", 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4000000000000001e-07 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564643+0000", 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0966e-05 2024-09-04T21:54:25.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564691+0000", 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8231e-05 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564714+0000", 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2857000000000002e-05 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.565011+0000", 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029731499999999998 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.565062+0000", 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1258000000000003e-05 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567253+0000", 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021909210000000002 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567369+0000", 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000115974 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.926 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7862 3.4 e32/22 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head v 32'76, mlcod=30'73)", 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.338481138999999, 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031099529999999999, 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566098+0000", 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566099+0000", 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9999999999999996e-07 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566110+0000", 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1009000000000001e-05 2024-09-04T21:54:25.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566141+0000", 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0244e-05 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566161+0000", 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0224000000000002e-05 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566407+0000", 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024641000000000002 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566463+0000", 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6113999999999998e-05 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569116+0000", 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026528519999999998 2024-09-04T21:54:25.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569208+0000", 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2299999999999994e-05 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14326 3.0 3:0619fb3a:::rbd_header.118816801d55:head [watch watch cookie 139952114223744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.2822391359999994, 2024-09-04T21:54:25.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00266585, 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14326 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622340+0000", 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622342+0000", 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622350+0000", 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0190000000000006e-06 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622373+0000", 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2634999999999999e-05 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622385+0000", 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622460+0000", 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4231000000000001e-05 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622524+0000", 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4548000000000006e-05 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.622642+0000", 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117707 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.623541+0000", 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00089878600000000005 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.624756+0000", 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012156300000000001 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.624853+0000", 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6292000000000006e-05 2024-09-04T21:54:25.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.625006+0000", 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015378000000000001 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14565 3.0 3:05912ce2:::rbd_data.118816801d55.00000000000000ee:head [sparse-read 0~524288] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.1828527659999999, 2024-09-04T21:54:25.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030052669999999998, 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14565 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:25.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721727+0000", 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721728+0000", 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0669999999999999e-06 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721733+0000", 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0189999999999997e-06 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721749+0000", 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6303e-05 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.721757+0000", 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7680000000000008e-06 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.723062+0000", 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001305283 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.724732+0000", 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001669827 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.941 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:14711 3.0 3:0619fb3a:::rbd_header.118816801d55:head [watch unwatch cookie 139952114223744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.0855729170000004, 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028282910000000001, 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 14711 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819007+0000", 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819010+0000", 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8569999999999999e-06 2024-09-04T21:54:25.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819020+0000", 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0808999999999999e-05 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819049+0000", 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8697999999999999e-05 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819061+0000", 2024-09-04T21:54:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1617000000000001e-05 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819171+0000", 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011018000000000001 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819302+0000", 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000130751 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.819493+0000", 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019150100000000001 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.820539+0000", 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010458620000000001 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821701+0000", 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011623009999999999 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821758+0000", 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6988999999999999e-05 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:16.821835+0000", 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6725999999999997e-05 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.947 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:21464 3.4 3:264f9539:::rbd_data.118892c64562.000000000000004f:head [set-alloc-hint object_size 2097152 write_size 2097152,write 465780~9356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.935034+0000", 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.9695457840000001, 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030008629999999999, 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 21464 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935034+0000", 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935034+0000", 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935036+0000", 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.159e-06 2024-09-04T21:54:25.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935049+0000", 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2575000000000001e-05 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935068+0000", 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9405000000000001e-05 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935078+0000", 2024-09-04T21:54:25.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0277e-05 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935153+0000", 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5220000000000001e-05 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935283+0000", 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012953699999999999 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.935385+0000", 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010186799999999999 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.936350+0000", 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00096469600000000002 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.937960+0000", 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001610301 2024-09-04T21:54:25.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.938002+0000", 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2364000000000003e-05 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.938035+0000", 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2461e-05 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:21751 3.4 3:264f9539:::rbd_data.118892c64562.000000000000004f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.001161+0000", 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.9034189379999997, 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038266440000000001, 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 21751 2024-09-04T21:54:25.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001161+0000", 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001161+0000", 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001163+0000", 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4250000000000001e-06 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001171+0000", 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9300000000000003e-06 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001193+0000", 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2103000000000002e-05 2024-09-04T21:54:25.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001204+0000", 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0969e-05 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001280+0000", 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6143000000000006e-05 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001378+0000", 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7761000000000001e-05 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.001496+0000", 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117997 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.002109+0000", 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000612618 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.004887+0000", 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027781720000000002 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.004943+0000", 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6069999999999997e-05 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.004987+0000", 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4456000000000003e-05 2024-09-04T21:54:25.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23212 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.316313+0000", 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.5882667350000004, 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.012575382, 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23212 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.316313+0000", 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.316313+0000", 2024-09-04T21:54:25.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.316315+0000", 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9589999999999998e-06 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317607+0000", 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012921689999999999 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317629+0000", 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.215e-05 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317638+0000", 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8380000000000008e-06 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317713+0000", 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5310999999999998e-05 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.317852+0000", 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013871399999999999 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.318878+0000", 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010260499999999999 2024-09-04T21:54:25.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.322933+0000", 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040546710000000001 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.328809+0000", 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058764170000000001 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.328853+0000", 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4162999999999997e-05 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.328888+0000", 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4940000000000001e-05 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23244 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [stat,write 868352~13015] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.385214+0000", 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.5193652440000003, 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031555229999999999, 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.967 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23244 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385214+0000", 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385214+0000", 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385216+0000", 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.655e-06 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385229+0000", 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.331e-05 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385242+0000", 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2906e-05 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385258+0000", 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5875e-05 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385324+0000", 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5587999999999996e-05 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385387+0000", 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3602000000000005e-05 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.385472+0000", 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4652000000000002e-05 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.386310+0000", 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083819500000000002 2024-09-04T21:54:25.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.388253+0000", 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019430610000000001 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.388321+0000", 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7583000000000007e-05 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.388370+0000", 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9095999999999999e-05 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23301 3.0 3:04b5afe4:::rbd_data.118892c64562.000000000000006a:head [set-alloc-hint object_size 2097152 write_size 2097152,write 638976~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.465580+0000", 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.4389998830000001, 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035173539999999999, 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.973 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23301 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465580+0000", 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465580+0000", 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465582+0000", 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465594+0000", 2024-09-04T21:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2432999999999999e-05 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465609+0000", 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4763000000000001e-05 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465615+0000", 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2179999999999996e-06 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465728+0000", 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011267 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.465952+0000", 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022462099999999999 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.466117+0000", 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016514700000000001 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.467064+0000", 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00094625899999999997 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.469009+0000", 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001945612 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.469059+0000", 2024-09-04T21:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9796000000000002e-05 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.469097+0000", 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7991000000000003e-05 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23323 3.0 3:04b5afe4:::rbd_data.118892c64562.000000000000006a:head [truncate 147456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.496920+0000", 2024-09-04T21:54:25.979 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.4076595239999996, 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026234370000000002, 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23323 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496920+0000", 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496920+0000", 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496923+0000", 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0639999999999998e-06 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496933+0000", 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496955+0000", 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2583e-05 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.496964+0000", 2024-09-04T21:54:25.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7930000000000003e-06 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.497078+0000", 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000114144 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.497203+0000", 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012433499999999999 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.497378+0000", 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017547099999999999 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.498251+0000", 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087273400000000001 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.499456+0000", 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012056809999999999 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.499491+0000", 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4390000000000001e-05 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.499544+0000", 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2741999999999997e-05 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23513 3.0 3:04b5afe4:::rbd_data.118892c64562.000000000000006a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.580224+0000", 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.3243560309999998, 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026765420000000001, 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23513 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580224+0000", 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580224+0000", 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580225+0000", 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6589999999999999e-06 2024-09-04T21:54:25.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580234+0000", 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0829999999999993e-06 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580250+0000", 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6127e-05 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580259+0000", 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2579999999999995e-06 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580336+0000", 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7667e-05 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580434+0000", 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7844999999999996e-05 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.580571+0000", 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013706999999999999 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.581099+0000", 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00052805199999999999 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.582819+0000", 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017201320000000001 2024-09-04T21:54:25.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.582855+0000", 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.54e-05 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.582900+0000", 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5249e-05 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23649 3.4 3:264f9539:::rbd_data.118892c64562.000000000000004f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.716257+0000", 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.1883229320000002, 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029682300000000001, 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.992 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23649 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716257+0000", 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716257+0000", 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716259+0000", 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9300000000000002e-06 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716265+0000", 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1360000000000002e-06 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716282+0000", 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6816e-05 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716290+0000", 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0790000000000001e-06 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716361+0000", 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0894000000000003e-05 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716451+0000", 2024-09-04T21:54:25.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0403999999999997e-05 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.716563+0000", 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011253500000000001 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.717130+0000", 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056608899999999998 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.719109+0000", 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019795469999999999 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.719175+0000", 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5440999999999999e-05 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.719225+0000", 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0358999999999997e-05 2024-09-04T21:54:25.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:23918 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [stat,truncate 1622016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.726035+0000", 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.1785444580000002, 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032032509999999998, 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23918 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726035+0000", 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726035+0000", 2024-09-04T21:54:25.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726037+0000", 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0140000000000001e-06 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726045+0000", 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3209999999999999e-06 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726063+0000", 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8875000000000001e-05 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726073+0000", 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3409999999999992e-06 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726154+0000", 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1396999999999993e-05 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726264+0000", 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010967300000000001 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.726426+0000", 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000162275 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.727165+0000", 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00073903399999999996 2024-09-04T21:54:26.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.729124+0000", 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019587480000000002 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.729191+0000", 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6913000000000001e-05 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.729238+0000", 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7660000000000001e-05 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:24093 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [stat,truncate 1540096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:21.990119+0000", 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.9144609799999999, 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041675929999999998, 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:26.004 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 24093 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990119+0000", 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990119+0000", 2024-09-04T21:54:26.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990121+0000", 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.756e-06 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990133+0000", 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1440000000000001e-05 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990159+0000", 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6264000000000001e-05 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990170+0000", 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1309e-05 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990246+0000", 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5522e-05 2024-09-04T21:54:26.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990338+0000", 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2435999999999994e-05 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.990478+0000", 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014000800000000001 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:26.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.991335+0000", 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085667000000000002 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.994205+0000", 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028697499999999999 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.994239+0000", 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3714999999999998e-05 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:21.994286+0000", 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7722999999999997e-05 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:24378 3.0 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head [stat,truncate 1081344] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:22.685289+0000", 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.2192905629999999, 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0070927259999999997, 2024-09-04T21:54:26.010 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 24378 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685289+0000", 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685289+0000", 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685292+0000", 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5569999999999998e-06 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685300+0000", 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9719999999999992e-06 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685320+0000", 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9958000000000001e-05 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685330+0000", 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0857e-05 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685420+0000", 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9448000000000001e-05 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685504+0000", 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4102000000000002e-05 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.685682+0000", 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000178099 2024-09-04T21:54:26.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.687224+0000", 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001542177 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.692311+0000", 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0050872039999999997 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:26.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.692351+0000", 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9746000000000002e-05 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:22.692382+0000", 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0605999999999998e-05 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:26381 3.1a e44/35 3:58784609:::rbd_data.118852763fbb.0000000000000001:head v 44'142, mlcod=43'140)", 2024-09-04T21:54:26.016 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:25.580026+0000", 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.32455372599999999, 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0062344569999999997, 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.580026+0000", 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.580026+0000", 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.580028+0000", 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.637e-06 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.585116+0000", 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0050882519999999997 2024-09-04T21:54:26.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.585149+0000", 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3584999999999998e-05 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.585165+0000", 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.585278+0000", 2024-09-04T21:54:26.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011320299999999999 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.585318+0000", 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0250999999999998e-05 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.586228+0000", 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090985600000000001 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:25.586260+0000", 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2211e-05 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.021 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:26.023 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2206 writesame 0x163ebda thru 0x1641ded data_size 0xa(0x3214 bytes) 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2208 read 0xcc3469 thru 0xccaa9b (0x7633 bytes) 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2210 punch from 0x17158a1 to 0x1718cfd, (0x345c bytes) 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2214 read 0x5e46 thru 0x8c8b (0x2e46 bytes) 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2215 read 0x16db00f thru 0x16e283b (0x782d bytes) 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2216 writesame 0x7be8dc9 thru 0x7bf2c1e data_size 0xd(0x9e56 bytes) 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2217 trunc from 0x7bf2c1f to 0x585d6fd 2024-09-04T21:54:26.024 INFO:teuthology.orchestra.run.smithi112.stdout:2219 trunc from 0x585d6fd to 0xb35e98b 2024-09-04T21:54:26.025 INFO:teuthology.orchestra.run.smithi112.stdout:2220 trunc from 0xb35e98b to 0x538b05b 2024-09-04T21:54:26.025 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:26.018+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:26.042 INFO:teuthology.orchestra.run.smithi112.stdout:2222 write 0x94257d2 thru 0x942a482 (0x4cb1 bytes) 2024-09-04T21:54:26.046 INFO:teuthology.orchestra.run.smithi112.stdout:2223 write 0x237978 thru 0x245ff7 (0xe680 bytes) 2024-09-04T21:54:26.050 INFO:teuthology.orchestra.run.smithi112.stdout:2224 read 0x83682fb thru 0x836e86f (0x6575 bytes) 2024-09-04T21:54:26.076 INFO:teuthology.orchestra.run.smithi112.stdout:2225 write 0xdb5e00a thru 0xdb6d7ea (0xf7e1 bytes) 2024-09-04T21:54:26.095 INFO:teuthology.orchestra.run.smithi112.stdout:2227 write 0xecf21c7 thru 0xecf23ba (0x1f4 bytes) 2024-09-04T21:54:26.101 INFO:teuthology.orchestra.run.smithi112.stdout:2228 write 0x6aa1755 thru 0x6aa9e27 (0x86d3 bytes) 2024-09-04T21:54:26.104 INFO:teuthology.orchestra.run.smithi112.stdout:2229 punch from 0x2aa2e78 to 0x2aaf109, (0xc291 bytes) 2024-09-04T21:54:26.104 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:26.104 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:26.104 INFO:teuthology.orchestra.run.smithi112.stdout:2233 read 0x367a135 thru 0x367f89a (0x5766 bytes) 2024-09-04T21:54:26.105 INFO:teuthology.orchestra.run.smithi112.stdout:2234 read 0x15a8176 thru 0x15b53c5 (0xd250 bytes) 2024-09-04T21:54:26.105 INFO:teuthology.orchestra.run.smithi112.stdout:2235 read 0x22696c5 thru 0x2279444 (0xfd80 bytes) 2024-09-04T21:54:26.106 INFO:teuthology.orchestra.run.smithi112.stdout:2236 writesame 0x97a58af thru 0x97b2cf3 data_size 0x7(0xd445 bytes) 2024-09-04T21:54:26.117 INFO:teuthology.orchestra.run.smithi112.stdout:2240 write 0x59510fc thru 0x5960cb6 (0xfbbb bytes) 2024-09-04T21:54:26.121 INFO:teuthology.orchestra.run.smithi112.stdout:2241 punch from 0xf1bc78 to 0xf27175, (0xb4fd bytes) 2024-09-04T21:54:26.123 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:26.118+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:26.134 INFO:teuthology.orchestra.run.smithi112.stdout:2244 trunc from 0xecf23bb to 0x6e0ec85 2024-09-04T21:54:26.161 INFO:teuthology.orchestra.run.smithi112.stdout:2245 write 0x44764af thru 0x4479e98 (0x39ea bytes) 2024-09-04T21:54:26.163 INFO:teuthology.orchestra.run.smithi112.stdout:2246 read 0x3025fa7 thru 0x303597e (0xf9d8 bytes) 2024-09-04T21:54:26.164 INFO:teuthology.orchestra.run.smithi112.stdout:2247 write 0x43f88d6 thru 0x44066b1 (0xdddc bytes) 2024-09-04T21:54:26.178 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:26.178 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:26.179 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:26.179 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:26.189 INFO:teuthology.orchestra.run.smithi112.stdout:2249 write 0xd685938 thru 0xd68e5e4 (0x8cad bytes) 2024-09-04T21:54:26.194 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:26.211 INFO:teuthology.orchestra.run.smithi112.stdout:2250 trunc from 0xd68e5e5 to 0x3b26758 2024-09-04T21:54:26.223 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:26.222+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:26.226 INFO:teuthology.orchestra.run.smithi112.stdout:2251 punch from 0x2e08ea3 to 0x2e11618, (0x8775 bytes) 2024-09-04T21:54:26.228 INFO:teuthology.orchestra.run.smithi112.stdout:2252 trunc from 0x3b26758 to 0x4dbf6a1 2024-09-04T21:54:26.239 INFO:teuthology.orchestra.run.smithi112.stdout:2253 write 0x389b5ae thru 0x38aac1d (0xf670 bytes) 2024-09-04T21:54:26.242 INFO:teuthology.orchestra.run.smithi112.stdout:2254 punch from 0x1b468ce to 0x1b5074b, (0x9e7d bytes) 2024-09-04T21:54:26.242 INFO:teuthology.orchestra.run.smithi112.stdout:2255 read 0x30b9dff thru 0x30c4b01 (0xad03 bytes) 2024-09-04T21:54:26.243 INFO:teuthology.orchestra.run.smithi112.stdout:2256 read 0x1f04b1a thru 0x1f0a625 (0x5b0c bytes) 2024-09-04T21:54:26.243 INFO:teuthology.orchestra.run.smithi112.stdout:2260 clone 12 order 23 su 32768 sc 4 2024-09-04T21:54:26.324 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:26.322+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:26.334 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:26.334 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:26.348 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:26.436 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:26.434+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:26.512 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7369 3.2 3:473dc813:::rbd_data.1188b1294ada.0000000000000008:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.393710253999998, 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035947, 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7369 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117106+0000", 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9799999999999998e-06 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117118+0000", 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1792e-05 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117149+0000", 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0412999999999998e-05 2024-09-04T21:54:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117163+0000", 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4779e-05 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117282+0000", 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011867 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117441+0000", 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015941300000000001 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.123847+0000", 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064053850000000004 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152852+0000", 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02900525 2024-09-04T21:54:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152883+0000", 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0756999999999997e-05 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152973+0000", 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0002999999999999e-05 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.153050+0000", 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7558000000000005e-05 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7714 3.7 3:e20eb1bd:::rbd_data.1188b1294ada.0000000000000030:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.678995195999999, 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034647877000000001, 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7714 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831822+0000", 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1599999999999998e-06 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831833+0000", 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1472e-05 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831865+0000", 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1707e-05 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831880+0000", 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4871e-05 2024-09-04T21:54:26.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.832016+0000", 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000136685 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.832254+0000", 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023754099999999999 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:26.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866177+0000", 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.03392299 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866313+0000", 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135995 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866466+0000", 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000153456 2024-09-04T21:54:26.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20788 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 11993088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7892099100000003, 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024856677000000001, 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.525 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20788 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:26.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721604+0000", 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2399999999999998e-07 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721606+0000", 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3010000000000001e-06 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721613+0000", 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3679999999999998e-06 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721618+0000", 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8729999999999998e-06 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746259+0000", 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024641237 2024-09-04T21:54:26.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746343+0000", 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3745000000000005e-05 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746460+0000", 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011762899999999999 2024-09-04T21:54:26.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20789 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12156928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7891897569999999, 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025041028, 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20789 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721624+0000", 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7300000000000002e-07 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721626+0000", 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7460000000000001e-06 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721631+0000", 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5230000000000003e-06 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721636+0000", 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.933e-06 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746494+0000", 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024857240999999999 2024-09-04T21:54:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746575+0000", 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1833000000000001e-05 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746665+0000", 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9378999999999999e-05 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20790 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12320768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721645+0000", 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7891683570000003, 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025214770000000001, 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.538 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20790 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721645+0000", 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721645+0000", 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721646+0000", 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8299999999999997e-07 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721648+0000", 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3609999999999999e-06 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721655+0000", 2024-09-04T21:54:26.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8730000000000001e-06 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721660+0000", 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1889999999999998e-06 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746696+0000", 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025036202 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746781+0000", 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4690000000000004e-05 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746860+0000", 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8972000000000002e-05 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20791 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12566528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.542 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.789145972, 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025394371999999998, 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20791 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721668+0000", 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721674+0000", 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4250000000000003e-06 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721681+0000", 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9500000000000004e-06 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721688+0000", 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4420000000000003e-06 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746892+0000", 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02520441 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.746978+0000", 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5370000000000004e-05 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747062+0000", 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4321999999999997e-05 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20792 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12730368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721696+0000", 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7891175810000002, 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025565674, 2024-09-04T21:54:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20792 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721696+0000", 2024-09-04T21:54:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721696+0000", 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721697+0000", 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2799999999999995e-07 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721699+0000", 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5320000000000001e-06 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721706+0000", 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.759e-06 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721711+0000", 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4280000000000004e-06 2024-09-04T21:54:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747092+0000", 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02538033 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747178+0000", 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5801000000000001e-05 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747262+0000", 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4096000000000003e-05 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20793 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 12894208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721718+0000", 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7890952359999996, 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025730287000000001, 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.552 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20793 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721718+0000", 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721718+0000", 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721719+0000", 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-07 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721721+0000", 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2230000000000001e-06 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721727+0000", 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4139999999999999e-06 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721733+0000", 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:54:26.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747295+0000", 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025562135 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747371+0000", 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5903999999999997e-05 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747449+0000", 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7955999999999997e-05 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20794 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7890740560000005, 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025923188, 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.557 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20794 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721740+0000", 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7e-07 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721743+0000", 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6469999999999998e-06 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721749+0000", 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6810000000000001e-06 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721755+0000", 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2549999999999997e-06 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747505+0000", 2024-09-04T21:54:26.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025750287 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747581+0000", 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5967e-05 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747663+0000", 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1880999999999998e-05 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20795 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7890517519999998, 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026093022, 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.562 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20795 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721762+0000", 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6800000000000001e-07 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721764+0000", 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721771+0000", 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6069999999999996e-06 2024-09-04T21:54:26.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721776+0000", 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9030000000000003e-06 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747695+0000", 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025918773999999999 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747777+0000", 2024-09-04T21:54:26.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2089000000000006e-05 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747855+0000", 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8118e-05 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20796 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13631488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721796+0000", 2024-09-04T21:54:26.566 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7890175230000001, 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026249567000000001, 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20796 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721796+0000", 2024-09-04T21:54:26.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721796+0000", 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721797+0000", 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2699999999999999e-07 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721799+0000", 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4229999999999999e-06 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721807+0000", 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721812+0000", 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5539999999999997e-06 2024-09-04T21:54:26.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747888+0000", 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026075528000000001 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.747970+0000", 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2181000000000005e-05 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748046+0000", 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5438000000000005e-05 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20797 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 13877248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7889930400000003, 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026421463999999999, 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.571 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20797 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721821+0000", 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7899999999999998e-07 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721823+0000", 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1780000000000002e-06 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721830+0000", 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5270000000000004e-06 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721839+0000", 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3479999999999993e-06 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748079+0000", 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026239357000000001 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748164+0000", 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4982999999999996e-05 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748242+0000", 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8492000000000007e-05 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20798 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14041088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.788966727, 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026571786999999999, 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20798 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.576 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721847+0000", 2024-09-04T21:54:26.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6199999999999998e-07 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721849+0000", 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1430000000000001e-06 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721856+0000", 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6159999999999998e-06 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721861+0000", 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0420000000000002e-06 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748273+0000", 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026411453000000001 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748352+0000", 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9412000000000005e-05 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748419+0000", 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6659e-05 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20799 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14204928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7889450079999998, 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027941948000000001, 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.580 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20799 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721869+0000", 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.82e-07 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721871+0000", 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1859999999999999e-06 2024-09-04T21:54:26.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721878+0000", 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6329999999999999e-06 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.721883+0000", 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0459999999999996e-06 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748452+0000", 2024-09-04T21:54:26.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.026568852 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748577+0000", 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012479599999999999 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749811+0000", 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012339530000000001 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20800 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14614528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723614+0000", 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7871999619999999, 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025236848999999999, 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20800 2024-09-04T21:54:26.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723614+0000", 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723614+0000", 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723615+0000", 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.209e-06 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723620+0000", 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1989999999999997e-06 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723635+0000", 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723643+0000", 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1659999999999994e-06 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748682+0000", 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025039446 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748771+0000", 2024-09-04T21:54:26.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.878e-05 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748850+0000", 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9561000000000006e-05 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20801 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14778368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723740+0000", 2024-09-04T21:54:26.589 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7870735570000003, 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02527865, 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20801 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723740+0000", 2024-09-04T21:54:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723740+0000", 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723741+0000", 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3399999999999998e-07 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723744+0000", 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7079999999999999e-06 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723759+0000", 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4364e-05 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723767+0000", 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-04T21:54:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748879+0000", 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025112025999999999 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.748965+0000", 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5673000000000005e-05 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749019+0000", 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3523000000000002e-05 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20802 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 14942208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7870340059999998, 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025380264999999999, 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20802 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723780+0000", 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723783+0000", 2024-09-04T21:54:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8629999999999999e-06 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723791+0000", 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6550000000000004e-06 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723796+0000", 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7520000000000002e-06 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749043+0000", 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025246951 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749099+0000", 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5179000000000001e-05 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749160+0000", 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1241999999999994e-05 2024-09-04T21:54:26.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20803 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 15187968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723805+0000", 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7870085539999998, 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025468752000000001, 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20803 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723805+0000", 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723805+0000", 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723806+0000", 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4000000000000002e-07 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723808+0000", 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3429999999999998e-06 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723815+0000", 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9999999999999999e-06 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723867+0000", 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2061000000000002e-05 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749175+0000", 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025308130000000002 2024-09-04T21:54:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749218+0000", 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2945999999999998e-05 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749274+0000", 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5732000000000001e-05 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20804 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 15351808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723881+0000", 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.7869324200000003, 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0255727, 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20804 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723881+0000", 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723881+0000", 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723882+0000", 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5400000000000001e-07 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723885+0000", 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4809999999999998e-06 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723894+0000", 2024-09-04T21:54:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9900000000000003e-06 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723901+0000", 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3330000000000002e-06 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749302+0000", 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025401182000000001 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749378+0000", 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6360999999999996e-05 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749454+0000", 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5698999999999995e-05 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:20805 3.2 3:4739397d:::rbd_data.1188cf8351f7.0000000000000002:b [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.786903863, 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025892301, 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 20805 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723910+0000", 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723913+0000", 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8260000000000001e-06 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723923+0000", 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7159999999999998e-06 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.723929+0000", 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7370000000000003e-06 2024-09-04T21:54:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749483+0000", 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025553868 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749569+0000", 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6494000000000002e-05 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:20.749802+0000", 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023295999999999999 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:26.613 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:26.614 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:26.615 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:26.610+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:26.716 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:26.710+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:26.749 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:26.749 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:26.749 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:26.749 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:26.764 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:26.817 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:26.814+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:26.890 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:26.890 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:26.890 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:26.890 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:26.890 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:26.896 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-09-04T21:54:26.906 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:26.917 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:26.914+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:26.957 INFO:teuthology.orchestra.run.smithi112.stdout:2261 trunc from 0x4dbf6a1 to 0xdbc6e53 2024-09-04T21:54:26.974 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:26.974 INFO:teuthology.orchestra.run.smithi112.stdout:2264 writesame 0x91b39f thru 0x920cc0 data_size 0x3(0x5922 bytes) 2024-09-04T21:54:27.018 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:27.014+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:27.046 INFO:teuthology.orchestra.run.smithi112.stdout:2265 read 0xca0e870 thru 0xca1646e (0x7bff bytes) 2024-09-04T21:54:27.046 INFO:teuthology.orchestra.run.smithi112.stdout:2267 read 0x678d6c5 thru 0x679d578 (0xfeb4 bytes) 2024-09-04T21:54:27.046 INFO:teuthology.orchestra.run.smithi112.stdout:2269 read 0x3cd7eac thru 0x3ce73be (0xf513 bytes) 2024-09-04T21:54:27.047 INFO:teuthology.orchestra.run.smithi112.stdout:2270 writesame 0x5b05e82 thru 0x5b07518 data_size 0x1(0x1697 bytes) 2024-09-04T21:54:27.065 INFO:teuthology.orchestra.run.smithi112.stdout:2271 write 0x932333c thru 0x93306a4 (0xd369 bytes) 2024-09-04T21:54:27.071 INFO:teuthology.orchestra.run.smithi112.stdout:2273 read 0x40a41d4 thru 0x40b3a15 (0xf842 bytes) 2024-09-04T21:54:27.072 INFO:teuthology.orchestra.run.smithi112.stdout:2275 writesame 0x6aba46d thru 0x6abcd4e data_size 0x1(0x28e2 bytes) 2024-09-04T21:54:27.079 INFO:teuthology.orchestra.run.smithi112.stdout:2276 writesame 0x7030998 thru 0x7036e3a data_size 0x1(0x64a3 bytes) 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20599 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 12075008~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.094 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740633+0000", 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.352681284, 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048637477999999998, 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20599 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740633+0000", 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740633+0000", 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740634+0000", 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9699999999999995e-07 2024-09-04T21:54:27.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740637+0000", 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0529999999999998e-06 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740646+0000", 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5159999999999994e-06 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740653+0000", 2024-09-04T21:54:27.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1230000000000002e-06 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789019+0000", 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048366141000000001 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789112+0000", 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2668000000000001e-05 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789271+0000", 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015928000000000001 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20600 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 12484608~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740662+0000", 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3526528549999997, 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049055162999999999, 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.099 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20600 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740662+0000", 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740662+0000", 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740662+0000", 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2699999999999999e-07 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740665+0000", 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5959999999999998e-06 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740674+0000", 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7029999999999995e-06 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740681+0000", 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4030000000000001e-06 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789336+0000", 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048654810999999999 2024-09-04T21:54:27.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789508+0000", 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017247399999999999 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789717+0000", 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020854900000000001 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20601 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 12648448~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740690+0000", 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3526245069999998, 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049323612000000003, 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20601 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740690+0000", 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740690+0000", 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740691+0000", 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3e-07 2024-09-04T21:54:27.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740694+0000", 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9799999999999998e-06 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740702+0000", 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5939999999999994e-06 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740710+0000", 2024-09-04T21:54:27.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3019999999999997e-06 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789799+0000", 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049089806999999999 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.789926+0000", 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000126425 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790014+0000", 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7874000000000006e-05 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20602 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 13058048~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740719+0000", 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3525959710000004, 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049631644000000003, 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.108 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20602 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740719+0000", 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740719+0000", 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740720+0000", 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7000000000000003e-07 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740722+0000", 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.807e-06 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740733+0000", 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0362e-05 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740742+0000", 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8929999999999994e-06 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790077+0000", 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049335757000000001 2024-09-04T21:54:27.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790214+0000", 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000136365 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790350+0000", 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013648999999999999 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20603 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 13221888~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740751+0000", 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.352563623, 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049860608000000001, 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20603 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.113 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740751+0000", 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740751+0000", 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740752+0000", 2024-09-04T21:54:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3e-07 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740763+0000", 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1382999999999999e-05 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740773+0000", 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6860000000000001e-06 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740780+0000", 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4930000000000001e-06 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790420+0000", 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049639753000000002 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790522+0000", 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000101949 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790612+0000", 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9714000000000002e-05 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20604 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 13385728~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740792+0000", 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.352522478, 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050136673, 2024-09-04T21:54:27.117 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20604 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740792+0000", 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740792+0000", 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740793+0000", 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.364e-06 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740798+0000", 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6859999999999997e-06 2024-09-04T21:54:27.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740815+0000", 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6728999999999999e-05 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740828+0000", 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3132999999999999e-05 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790674+0000", 2024-09-04T21:54:27.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.049846335999999998 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790816+0000", 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014211200000000001 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790929+0000", 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011231299999999999 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20605 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 13795328~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740846+0000", 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3524689800000003, 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050313681999999998, 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20605 2024-09-04T21:54:27.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740846+0000", 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740846+0000", 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740847+0000", 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2759999999999999e-06 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740851+0000", 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4900000000000002e-06 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740868+0000", 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6109000000000001e-05 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740880+0000", 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2547e-05 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.790971+0000", 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050091449000000003 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791072+0000", 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100881 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791159+0000", 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6929999999999996e-05 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20606 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 13959168~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740897+0000", 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3524176560000001, 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050587590000000002, 2024-09-04T21:54:27.126 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20606 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740897+0000", 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740897+0000", 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740898+0000", 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0839999999999999e-06 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740903+0000", 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6260000000000003e-06 2024-09-04T21:54:27.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740922+0000", 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8865999999999999e-05 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740936+0000", 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4552000000000001e-05 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791218+0000", 2024-09-04T21:54:27.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050282238 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791356+0000", 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013795100000000001 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791485+0000", 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012827300000000001 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.130 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20607 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 14368768~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.740955+0000", 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3523598650000004, 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050778831000000003, 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20607 2024-09-04T21:54:27.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740955+0000", 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740955+0000", 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740956+0000", 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740961+0000", 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1499999999999998e-06 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740978+0000", 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6739e-05 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.740991+0000", 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3553000000000001e-05 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791561+0000", 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050569171000000003 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791647+0000", 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6749999999999994e-05 2024-09-04T21:54:27.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791734+0000", 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6231999999999997e-05 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20608 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 14532608~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.135 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.741009+0000", 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3523060510000002, 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050921618000000002, 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20608 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741009+0000", 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741009+0000", 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741010+0000", 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1039999999999999e-06 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741015+0000", 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.186e-06 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741032+0000", 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6722999999999999e-05 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741047+0000", 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5755000000000001e-05 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791773+0000", 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050726061000000003 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791851+0000", 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7140999999999999e-05 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.791930+0000", 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9647999999999994e-05 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.139 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20609 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 14696448~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.741066+0000", 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.352248887, 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051291873000000002, 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20609 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741066+0000", 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741066+0000", 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741067+0000", 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7300000000000004e-07 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741071+0000", 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7400000000000004e-06 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741089+0000", 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7085999999999999e-05 2024-09-04T21:54:27.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741101+0000", 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.261e-05 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792010+0000", 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050908510999999997 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792157+0000", 2024-09-04T21:54:27.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014762999999999999 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792358+0000", 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000200323 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20610 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 15106048~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.741119+0000", 2024-09-04T21:54:27.144 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.352195429, 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051416174000000002, 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20610 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741119+0000", 2024-09-04T21:54:27.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741119+0000", 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741120+0000", 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.19e-06 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741125+0000", 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9830000000000004e-06 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741142+0000", 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6892999999999999e-05 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741155+0000", 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3139000000000001e-05 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792386+0000", 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051230150000000002 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792458+0000", 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2947000000000005e-05 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792535+0000", 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6872000000000005e-05 2024-09-04T21:54:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20611 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 15269888~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.741175+0000", 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3521397549999996, 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051607462999999999, 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20611 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741175+0000", 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741175+0000", 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741176+0000", 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2999999999999999e-07 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741181+0000", 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.023e-06 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741198+0000", 2024-09-04T21:54:27.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6769000000000002e-05 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741211+0000", 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2969999999999999e-05 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792606+0000", 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051394996999999998 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792696+0000", 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0282000000000001e-05 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792782+0000", 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6491999999999998e-05 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.153 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20612 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 15679488~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.741228+0000", 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3520868940000002, 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051749904999999999, 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20612 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741228+0000", 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741228+0000", 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741229+0000", 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.19e-06 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741234+0000", 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0629999999999996e-06 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741251+0000", 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7272999999999999e-05 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.741264+0000", 2024-09-04T21:54:27.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3091999999999999e-05 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792822+0000", 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051557970000000002 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792900+0000", 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7968999999999998e-05 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.792978+0000", 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7348000000000006e-05 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20735 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 14385152~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.749265+0000", 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.344050041, 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048348840999999997, 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.158 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20735 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749265+0000", 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749265+0000", 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749266+0000", 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8899999999999998e-07 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749281+0000", 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5532999999999998e-05 2024-09-04T21:54:27.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749299+0000", 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7533e-05 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749312+0000", 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3489000000000001e-05 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797441+0000", 2024-09-04T21:54:27.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048128424000000003 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797525+0000", 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4946999999999998e-05 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797613+0000", 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7925999999999998e-05 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.162 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20736 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 14548992~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.749331+0000", 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3439834079999997, 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048501848, 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20736 2024-09-04T21:54:27.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749331+0000", 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749331+0000", 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749333+0000", 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7770000000000001e-06 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749339+0000", 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6760000000000004e-06 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749356+0000", 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7649e-05 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749370+0000", 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3869999999999999e-05 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797670+0000", 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048299531 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797753+0000", 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2863999999999998e-05 2024-09-04T21:54:27.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797833+0000", 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0481000000000004e-05 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20737 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 14712832~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.749425+0000", 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3438893959999998, 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048626870000000003, 2024-09-04T21:54:27.167 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20737 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749425+0000", 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749425+0000", 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749427+0000", 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7439999999999999e-06 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749433+0000", 2024-09-04T21:54:27.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4230000000000002e-06 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749452+0000", 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9096000000000001e-05 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749467+0000", 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4885000000000001e-05 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797882+0000", 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048414347000000003 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.797970+0000", 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8517999999999995e-05 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798052+0000", 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1856999999999999e-05 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.171 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20738 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 15122432~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.749489+0000", 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3438252520000002, 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048778655999999997, 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.172 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20738 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749489+0000", 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749489+0000", 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749491+0000", 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3430000000000001e-06 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749496+0000", 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3240000000000002e-06 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749513+0000", 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7065e-05 2024-09-04T21:54:27.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749528+0000", 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4572999999999999e-05 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798101+0000", 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048573553999999998 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798183+0000", 2024-09-04T21:54:27.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1578999999999999e-05 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798268+0000", 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5217999999999996e-05 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:20739 3.1a 3:5a393c8b:::rbd_data.1188cf8351f7.0000000000000005:b [read 15286272~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.749548+0000", 2024-09-04T21:54:27.176 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.3437666740000003, 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048940536, 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 20739 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749548+0000", 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.749548+0000", 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.755438+0000", 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0058903480000000001 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.755450+0000", 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.216e-05 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.755479+0000", 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8949999999999999e-05 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.755499+0000", 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9673e-05 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798324+0000", 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.042825177999999998 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798410+0000", 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6077999999999998e-05 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.798488+0000", 2024-09-04T21:54:27.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8149e-05 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26381 3.1a 3:58784609:::rbd_data.118852763fbb.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.559534+0000", 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.533780932, 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051176559000000003, 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:27.181 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26381 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559534+0000", 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559534+0000", 2024-09-04T21:54:27.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559536+0000", 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1100000000000001e-06 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559544+0000", 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7020000000000007e-06 2024-09-04T21:54:27.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559566+0000", 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1685999999999999e-05 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559584+0000", 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7773000000000001e-05 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559696+0000", 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112205 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.559831+0000", 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134719 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.569897+0000", 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010066604 2024-09-04T21:54:27.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.610506+0000", 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.040608645999999998 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.610537+0000", 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1238000000000001e-05 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.610623+0000", 2024-09-04T21:54:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5833000000000003e-05 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.610710+0000", 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7043e-05 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:27.187 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:27.188 INFO:teuthology.orchestra.run.smithi112.stdout:2277 trunc from 0xdbc6e53 to 0x240c4fd 2024-09-04T21:54:27.189 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:27.189 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:27.186+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:27.219 INFO:teuthology.orchestra.run.smithi112.stdout:2278 read 0xdf2c5f thru 0xdf4426 (0x17c8 bytes) 2024-09-04T21:54:27.220 INFO:teuthology.orchestra.run.smithi112.stdout:2279 read 0x202a7f thru 0x20f5d6 (0xcb58 bytes) 2024-09-04T21:54:27.221 INFO:teuthology.orchestra.run.smithi112.stdout:2280 write 0x16ebaf thru 0x170cd6 (0x2128 bytes) 2024-09-04T21:54:27.290 INFO:teuthology.orchestra.run.smithi112.stdout:2281 read 0x1a443fe thru 0x1a4574a (0x134d bytes) 2024-09-04T21:54:27.290 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:27.286+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:27.330 INFO:teuthology.orchestra.run.smithi112.stdout:2282 write 0xd6f3a77 thru 0xd702b48 (0xf0d2 bytes) 2024-09-04T21:54:27.339 INFO:teuthology.orchestra.run.smithi112.stdout:2283 punch from 0xaa528f9 to 0xaa61346, (0xea4d bytes) 2024-09-04T21:54:27.339 INFO:teuthology.orchestra.run.smithi112.stdout:2285 write 0x13509f6 thru 0x13573bd (0x69c8 bytes) 2024-09-04T21:54:27.342 INFO:teuthology.orchestra.run.smithi112.stdout:2286 punch from 0x4db2714 to 0x4db8ec0, (0x67ac bytes) 2024-09-04T21:54:27.342 INFO:teuthology.orchestra.run.smithi112.stdout:2288 read 0xc65d6e4 thru 0xc65ef4f (0x186c bytes) 2024-09-04T21:54:27.343 INFO:teuthology.orchestra.run.smithi112.stdout:2289 punch from 0x23b6f8c to 0x23c568b, (0xe6ff bytes) 2024-09-04T21:54:27.343 INFO:teuthology.orchestra.run.smithi112.stdout:2291 read 0x9a12bca thru 0x9a1aca5 (0x80dc bytes) 2024-09-04T21:54:27.343 INFO:teuthology.orchestra.run.smithi112.stdout:2293 punch from 0xd191830 to 0xd19b0c2, (0x9892 bytes) 2024-09-04T21:54:27.358 INFO:teuthology.orchestra.run.smithi112.stdout:2294 write 0xe1be0cc thru 0xe1cc6ea (0xe61f bytes) 2024-09-04T21:54:27.365 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:27.365 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:27.365 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:27.365 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:27.366 INFO:teuthology.orchestra.run.smithi112.stdout:2295 read 0x75bd04f thru 0x75c0199 (0x314b bytes) 2024-09-04T21:54:27.366 INFO:teuthology.orchestra.run.smithi112.stdout:2296 read 0x63af3c1 thru 0x63b302f (0x3c6f bytes) 2024-09-04T21:54:27.367 INFO:teuthology.orchestra.run.smithi112.stdout:2297 read 0x3f88255 thru 0x3f91954 (0x9700 bytes) 2024-09-04T21:54:27.367 INFO:teuthology.orchestra.run.smithi112.stdout:2298 read 0x1cf9044 thru 0x1cfa7c0 (0x177d bytes) 2024-09-04T21:54:27.369 INFO:teuthology.orchestra.run.smithi112.stdout:2299 write 0xb6570c5 thru 0xb65844f (0x138b bytes) 2024-09-04T21:54:27.375 INFO:teuthology.orchestra.run.smithi112.stdout:2300 write 0xd0294e2 thru 0xd02dff6 (0x4b15 bytes) 2024-09-04T21:54:27.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:27.381 INFO:teuthology.orchestra.run.smithi112.stdout:2302 read 0x5390414 thru 0x5398034 (0x7c21 bytes) 2024-09-04T21:54:27.391 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:27.386+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:27.412 INFO:teuthology.orchestra.run.smithi112.stdout:2304 trunc from 0xe1cc6eb to 0x2f52b6b 2024-09-04T21:54:27.419 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:27.433 INFO:teuthology.orchestra.run.smithi112.stdout:2305 punch from 0x1aa8ce3 to 0x1ab41aa, (0xb4c7 bytes) 2024-09-04T21:54:27.434 INFO:teuthology.orchestra.run.smithi112.stdout:2306 read 0x2ec572c thru 0x2ecd9d7 (0x82ac bytes) 2024-09-04T21:54:27.434 INFO:teuthology.orchestra.run.smithi112.stdout:2307 read 0x1201601 thru 0x1204a52 (0x3452 bytes) 2024-09-04T21:54:27.452 INFO:teuthology.orchestra.run.smithi112.stdout:2309 write 0x839987d thru 0x83a2b7f (0x9303 bytes) 2024-09-04T21:54:27.457 INFO:teuthology.orchestra.run.smithi112.stdout:2311 read 0x57f0439 thru 0x57f4bda (0x47a2 bytes) 2024-09-04T21:54:27.458 INFO:teuthology.orchestra.run.smithi112.stdout:2313 flatten 2024-09-04T21:54:27.466 INFO:teuthology.orchestra.run.smithi112.stdout:2314 punch from 0x4f62eb0 to 0x4f6caca, (0x9c1a bytes) 2024-09-04T21:54:27.466 INFO:teuthology.orchestra.run.smithi112.stdout:2315 read 0x41e17f3 thru 0x41e3e6a (0x2678 bytes) 2024-09-04T21:54:27.466 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.466 INFO:teuthology.orchestra.run.smithi112.stdout:2317 punch from 0x6cbcd2d to 0x6cbed52, (0x2025 bytes) 2024-09-04T21:54:27.479 INFO:teuthology.orchestra.run.smithi112.stdout:2318 write 0xb5a6e61 thru 0xb5a75dc (0x77c bytes) 2024-09-04T21:54:27.492 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:27.490+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:27.495 INFO:teuthology.orchestra.run.smithi112.stdout:2319 write 0xe0a9976 thru 0xe0af459 (0x5ae4 bytes) 2024-09-04T21:54:27.499 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.499 INFO:teuthology.orchestra.run.smithi112.stdout:2323 write 0xb94984e thru 0xb9510cb (0x787e bytes) 2024-09-04T21:54:27.503 INFO:teuthology.orchestra.run.smithi112.stdout:2325 read 0x7d702e5 thru 0x7d7a00a (0x9d26 bytes) 2024-09-04T21:54:27.503 INFO:teuthology.orchestra.run.smithi112.stdout:2326 write 0x7f740c1 thru 0x7f7a2dc (0x621c bytes) 2024-09-04T21:54:27.507 INFO:teuthology.orchestra.run.smithi112.stdout:2328 punch from 0xa94fe61 to 0xa951fdc, (0x217b bytes) 2024-09-04T21:54:27.507 INFO:teuthology.orchestra.run.smithi112.stdout:2329 punch from 0xc7018ac to 0xc70f329, (0xda7d bytes) 2024-09-04T21:54:27.507 INFO:teuthology.orchestra.run.smithi112.stdout:2330 write 0x8c01a69 thru 0x8c1101e (0xf5b6 bytes) 2024-09-04T21:54:27.511 INFO:teuthology.orchestra.run.smithi112.stdout:2331 read 0x713848e thru 0x7141cdc (0x984f bytes) 2024-09-04T21:54:27.523 INFO:teuthology.orchestra.run.smithi112.stdout:2332 trunc from 0xe0af45a to 0x62c2cb6 2024-09-04T21:54:27.540 INFO:teuthology.orchestra.run.smithi112.stdout:2333 write 0x55f108f thru 0x55f6e8d (0x5dff bytes) 2024-09-04T21:54:27.545 INFO:teuthology.orchestra.run.smithi112.stdout:2335 read 0x24ded81 thru 0x24eb450 (0xc6d0 bytes) 2024-09-04T21:54:27.545 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:27.545 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:27.545 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:27.545 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:27.545 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:27.554 INFO:teuthology.orchestra.run.smithi112.stdout:2337 writesame 0x68f1af5 thru 0x6901a4d data_size 0x1(0xff59 bytes) 2024-09-04T21:54:27.556 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.556 INFO:teuthology.orchestra.run.smithi112.stdout:2339 punch from 0x4c72cf8 to 0x4c792d9, (0x65e1 bytes) 2024-09-04T21:54:27.556 INFO:teuthology.orchestra.run.smithi112.stdout:2340 punch from 0x350fee5 to 0x3515811, (0x592c bytes) 2024-09-04T21:54:27.556 INFO:teuthology.orchestra.run.smithi112.stdout:2341 trunc from 0x6901a4e to 0x707a0dc 2024-09-04T21:54:27.561 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:27.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:54:27.577 INFO:teuthology.orchestra.run.smithi112.stdout:2342 write 0xb45f4c4 thru 0xb4603a1 (0xede bytes) 2024-09-04T21:54:27.580 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:27.583 INFO:teuthology.orchestra.run.smithi112.stdout:2343 writesame 0x6ce58fe thru 0x6cf206e data_size 0x5d(0xc771 bytes) 2024-09-04T21:54:27.588 INFO:teuthology.orchestra.run.smithi112.stdout:2344 read 0xaf0dd40 thru 0xaf110ea (0x33ab bytes) 2024-09-04T21:54:27.588 INFO:teuthology.orchestra.run.smithi112.stdout:2345 read 0x20b383a thru 0x20b6f81 (0x3748 bytes) 2024-09-04T21:54:27.588 INFO:teuthology.orchestra.run.smithi112.stdout:2346 read 0x1c1b993 thru 0x1c1f70b (0x3d79 bytes) 2024-09-04T21:54:27.599 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:27.599 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:27.600 INFO:teuthology.orchestra.run.smithi112.stdout:2347 write 0xe2228b4 thru 0xe22f1da (0xc927 bytes) 2024-09-04T21:54:27.602 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:27.602+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:27.606 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.607 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.607 INFO:teuthology.orchestra.run.smithi112.stdout:2352 read 0x3bcab2d thru 0x3bd7f15 (0xd3e9 bytes) 2024-09-04T21:54:27.618 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:27.618 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:27.621 INFO:teuthology.orchestra.run.smithi112.stdout:2353 trunc from 0xe22f1db to 0x5788c1b 2024-09-04T21:54:27.635 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.635 INFO:teuthology.orchestra.run.smithi112.stdout:2355 read 0x409b928 thru 0x40a6129 (0xa802 bytes) 2024-09-04T21:54:27.635 INFO:teuthology.orchestra.run.smithi112.stdout:2356 write 0x3cc25ae thru 0x3cc99d2 (0x7425 bytes) 2024-09-04T21:54:27.636 INFO:teuthology.orchestra.run.smithi112.stdout:2357 read 0x135206c thru 0x13580f7 (0x608c bytes) 2024-09-04T21:54:27.637 INFO:teuthology.orchestra.run.smithi112.stdout:2358 read 0x4682aa5 thru 0x4687e64 (0x53c0 bytes) 2024-09-04T21:54:27.638 INFO:teuthology.orchestra.run.smithi112.stdout:2359 read 0x41e9813 thru 0x41ee580 (0x4d6e bytes) 2024-09-04T21:54:27.638 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:27.638 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:27.645 INFO:teuthology.orchestra.run.smithi112.stdout:2360 trunc from 0x5788c1b to 0x136b912 2024-09-04T21:54:27.658 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:27.658 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:27.659 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.659 INFO:teuthology.orchestra.run.smithi112.stdout:2362 read 0x712b4a thru 0x71759f (0x4a56 bytes) 2024-09-04T21:54:27.671 INFO:teuthology.orchestra.run.smithi112.stdout:2363 trunc from 0x136b912 to 0x7c804c0 2024-09-04T21:54:27.677 INFO:teuthology.orchestra.run.smithi112.stdout:2364 read 0x4463e58 thru 0x446bddf (0x7f88 bytes) 2024-09-04T21:54:27.678 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:27.678 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:27.691 INFO:teuthology.orchestra.run.smithi112.stdout:2365 writesame 0xbc61b9d thru 0xbc6e785 data_size 0x1(0xcbe9 bytes) 2024-09-04T21:54:27.692 INFO:teuthology.orchestra.run.smithi112.stdout:2367 trunc from 0xbc6e786 to 0xb6cdd7b 2024-09-04T21:54:27.693 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:27.690+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:27.698 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:27.699 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:27.699 INFO:teuthology.orchestra.run.smithi112.stdout:2369 read 0x6976824 thru 0x697edac (0x8589 bytes) 2024-09-04T21:54:27.699 INFO:teuthology.orchestra.run.smithi112.stdout:2370 write 0xa3cf698 thru 0xa3d1fe4 (0x294d bytes) 2024-09-04T21:54:27.699 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:27.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:27.705 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.705 INFO:teuthology.orchestra.run.smithi112.stdout:2372 writesame 0xaee09e thru 0xaf72f4 data_size 0x1(0x9257 bytes) 2024-09-04T21:54:27.729 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:27.729 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:27.733 INFO:teuthology.orchestra.run.smithi112.stdout:2373 write 0x1a2edd6 thru 0x1a3d588 (0xe7b3 bytes) 2024-09-04T21:54:27.735 INFO:teuthology.orchestra.run.smithi112.stdout:2374 write 0x5e6e428 thru 0x5e72783 (0x435c bytes) 2024-09-04T21:54:27.739 INFO:teuthology.orchestra.run.smithi112.stdout:2375 read 0x6e02b6d thru 0x6e0989a (0x6d2e bytes) 2024-09-04T21:54:27.745 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:27.759 INFO:teuthology.orchestra.run.smithi112.stdout:2377 trunc from 0xb6cdd7b to 0x55fa7e 2024-09-04T21:54:27.775 INFO:teuthology.orchestra.run.smithi112.stdout:2378 read 0x49ab3b thru 0x4a8813 (0xdcd9 bytes) 2024-09-04T21:54:27.795 INFO:teuthology.orchestra.run.smithi112.stdout:2379 trunc from 0x55fa7e to 0xb29a30b 2024-09-04T21:54:27.804 INFO:teuthology.orchestra.run.smithi112.stdout:2380 write 0x1a7e30a thru 0x1a835d3 (0x52ca bytes) 2024-09-04T21:54:27.807 INFO:teuthology.orchestra.run.smithi112.stdout:2381 trunc from 0xb29a30b to 0xad579fd 2024-09-04T21:54:27.818 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.818 INFO:teuthology.orchestra.run.smithi112.stdout:2383 read 0x6faf003 thru 0x6fbdac0 (0xeabe bytes) 2024-09-04T21:54:27.818 INFO:teuthology.orchestra.run.smithi112.stdout:2384 read 0x3ccdd2 thru 0x3d6a20 (0x9c4f bytes) 2024-09-04T21:54:27.819 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.819 INFO:teuthology.orchestra.run.smithi112.stdout:2386 write 0x1faa494 thru 0x1fb1248 (0x6db5 bytes) 2024-09-04T21:54:27.822 INFO:teuthology.orchestra.run.smithi112.stdout:2387 read 0x7f7e8d7 thru 0x7f83bd2 (0x52fc bytes) 2024-09-04T21:54:27.822 INFO:teuthology.orchestra.run.smithi112.stdout:2388 punch from 0x3f89fdd to 0x3f97f54, (0xdf77 bytes) 2024-09-04T21:54:27.822 INFO:teuthology.orchestra.run.smithi112.stdout:2389 write 0x1d8450e thru 0x1d8e427 (0x9f1a bytes) 2024-09-04T21:54:27.825 INFO:teuthology.orchestra.run.smithi112.stdout:2390 write 0x5e4083d thru 0x5e4d98a (0xd14e bytes) 2024-09-04T21:54:27.831 INFO:teuthology.orchestra.run.smithi112.stdout:2391 punch from 0x6519b1a to 0x6522b33, (0x9019 bytes) 2024-09-04T21:54:27.831 INFO:teuthology.orchestra.run.smithi112.stdout:2393 punch from 0x2eafc51 to 0x2eb5890, (0x5c3f bytes) 2024-09-04T21:54:27.831 INFO:teuthology.orchestra.run.smithi112.stdout:2394 punch from 0xa8f2df to 0xa9315d, (0x3e7e bytes) 2024-09-04T21:54:27.831 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.831 INFO:teuthology.orchestra.run.smithi112.stdout:2397 read 0x861fb1b thru 0x862bb49 (0xc02f bytes) 2024-09-04T21:54:27.832 INFO:teuthology.orchestra.run.smithi112.stdout:2399 punch from 0x3ba9f59 to 0x3bb0952, (0x69f9 bytes) 2024-09-04T21:54:27.832 INFO:teuthology.orchestra.run.smithi112.stdout:2400 write 0x740c59 thru 0x745e2b (0x51d3 bytes) 2024-09-04T21:54:27.833 INFO:teuthology.orchestra.run.smithi112.stdout:2402 read 0x9cd9bc0 thru 0x9cdc3e9 (0x282a bytes) 2024-09-04T21:54:27.833 INFO:teuthology.orchestra.run.smithi112.stdout:2403 punch from 0xad0f3b0 to 0xad12dd1, (0x3a21 bytes) 2024-09-04T21:54:27.834 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:27.834+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:27.846 INFO:teuthology.orchestra.run.smithi112.stdout:2404 write 0xdb1dc0e thru 0xdb2519d (0x7590 bytes) 2024-09-04T21:54:27.851 INFO:teuthology.orchestra.run.smithi112.stdout:2405 write 0x1c933f5 thru 0x1c99456 (0x6062 bytes) 2024-09-04T21:54:27.853 INFO:teuthology.orchestra.run.smithi112.stdout:2406 write 0x186c46e thru 0x1870436 (0x3fc9 bytes) 2024-09-04T21:54:27.855 INFO:teuthology.orchestra.run.smithi112.stdout:2407 punch from 0x1a98b5e to 0x1a9c2bf, (0x3761 bytes) 2024-09-04T21:54:27.855 INFO:teuthology.orchestra.run.smithi112.stdout:2410 read 0x4fe9b3d thru 0x4ff9acb (0xff8f bytes) 2024-09-04T21:54:27.860 INFO:teuthology.orchestra.run.smithi112.stdout:2412 trunc from 0xdb2519e to 0xb70422c 2024-09-04T21:54:27.875 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:27.890 INFO:teuthology.orchestra.run.smithi112.stdout:2414 writesame 0xe152e6f thru 0xe1562a5 data_size 0x1(0x3437 bytes) 2024-09-04T21:54:27.890 INFO:teuthology.orchestra.run.smithi112.stdout:2415 writesame 0x8173552 thru 0x81800e1 data_size 0x8(0xcb90 bytes) 2024-09-04T21:54:27.895 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:27.890+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:27.901 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:27.909 INFO:teuthology.orchestra.run.smithi112.stdout:2417 trunc from 0xe1562a6 to 0xb79115e 2024-09-04T21:54:27.916 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:27.916 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:27.920 INFO:teuthology.orchestra.run.smithi112.stdout:2418 read 0x71ccead thru 0x71d52d9 (0x842d bytes) 2024-09-04T21:54:27.920 INFO:teuthology.orchestra.run.smithi112.stdout:2420 writesame 0x9f3983 thru 0x9ff9ac data_size 0x1b(0xc02a bytes) 2024-09-04T21:54:27.924 INFO:teuthology.orchestra.run.smithi112.stdout:2421 read 0x8608671 thru 0x86120a6 (0x9a36 bytes) 2024-09-04T21:54:27.925 INFO:teuthology.orchestra.run.smithi112.stdout:2423 read 0x22645fa thru 0x226c0f7 (0x7afe bytes) 2024-09-04T21:54:27.925 INFO:teuthology.orchestra.run.smithi112.stdout:2425 punch from 0x5be4397 to 0x5beae30, (0x6a99 bytes) 2024-09-04T21:54:27.925 INFO:teuthology.orchestra.run.smithi112.stdout:2426 read 0xafebce5 thru 0xaff5a93 (0x9daf bytes) 2024-09-04T21:54:27.932 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:27.935 INFO:teuthology.orchestra.run.smithi112.stdout:2427 trunc from 0xb79115e to 0x7065863 2024-09-04T21:54:27.960 INFO:teuthology.orchestra.run.smithi112.stdout:2428 writesame 0x7731196 thru 0x773dea9 data_size 0x32(0xcd14 bytes) 2024-09-04T21:54:27.973 INFO:teuthology.orchestra.run.smithi112.stdout:2429 write 0x91fac3d thru 0x91fec04 (0x3fc8 bytes) 2024-09-04T21:54:27.978 INFO:teuthology.orchestra.run.smithi112.stdout:2430 read 0x56e4a81 thru 0x56e660b (0x1b8b bytes) 2024-09-04T21:54:27.979 INFO:teuthology.orchestra.run.smithi112.stdout:2432 writesame 0x19b35e5 thru 0x19bbc50 data_size 0xe(0x866c bytes) 2024-09-04T21:54:27.982 INFO:teuthology.orchestra.run.smithi112.stdout:2433 write 0x39a0c86 thru 0x39a3070 (0x23eb bytes) 2024-09-04T21:54:27.986 INFO:teuthology.orchestra.run.smithi112.stdout:2435 write 0x8b6ab9 thru 0x8bcab4 (0x5ffc bytes) 2024-09-04T21:54:27.988 INFO:teuthology.orchestra.run.smithi112.stdout:2436 punch from 0x47eedaa to 0x47fe691, (0xf8e7 bytes) 2024-09-04T21:54:27.989 INFO:teuthology.orchestra.run.smithi112.stdout:2437 read 0x59e71ac thru 0x59eebb9 (0x7a0e bytes) 2024-09-04T21:54:27.989 INFO:teuthology.orchestra.run.smithi112.stdout:2438 read 0x524e029 thru 0x5254ff1 (0x6fc9 bytes) 2024-09-04T21:54:28.010 INFO:teuthology.orchestra.run.smithi112.stdout:2439 write 0xdb1b22d thru 0xdb27e1e (0xcbf2 bytes) 2024-09-04T21:54:28.017 INFO:teuthology.orchestra.run.smithi112.stdout:2440 write 0x49e635 thru 0x4ae288 (0xfc54 bytes) 2024-09-04T21:54:28.020 INFO:teuthology.orchestra.run.smithi157.stderr:no valid command found; 10 closest matches: 2024-09-04T21:54:28.020 INFO:teuthology.orchestra.run.smithi157.stderr:0 2024-09-04T21:54:28.020 INFO:teuthology.orchestra.run.smithi157.stderr:1 2024-09-04T21:54:28.020 INFO:teuthology.orchestra.run.smithi157.stderr:2 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:abort 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:assert 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:config diff 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:config diff get 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:config get 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:config help [] 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:config set ... 2024-09-04T21:54:28.021 INFO:teuthology.orchestra.run.smithi157.stderr:admin_socket: invalid command 2024-09-04T21:54:28.021 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:28.018+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:28.022 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.022 INFO:teuthology.orchestra.run.smithi112.stdout:2442 write 0x96cf303 thru 0x96d096b (0x1669 bytes) 2024-09-04T21:54:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:54:28.023 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-04T21:54:28.025 INFO:teuthology.orchestra.run.smithi112.stdout:2443 read 0x227f08f thru 0x22835d3 (0x4545 bytes) 2024-09-04T21:54:28.026 INFO:teuthology.orchestra.run.smithi112.stdout:2444 write 0x2446fda thru 0x245433a (0xd361 bytes) 2024-09-04T21:54:28.055 INFO:teuthology.orchestra.run.smithi112.stdout:2445 trunc from 0xdb27e1f to 0x4e0545f 2024-09-04T21:54:28.094 INFO:teuthology.orchestra.run.smithi112.stdout:2446 writesame 0x699a813 thru 0x69a54b4 data_size 0x13(0xaca2 bytes) 2024-09-04T21:54:28.097 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:28.094+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:28.103 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.103 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:28.117 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:28.125 INFO:teuthology.orchestra.run.smithi112.stdout:2447 writesame 0xdaa02d7 thru 0xdaa157c data_size 0xe(0x12a6 bytes) 2024-09-04T21:54:28.125 INFO:teuthology.orchestra.run.smithi112.stdout:2448 writesame 0xd801755 thru 0xd8048c4 data_size 0x38(0x3170 bytes) 2024-09-04T21:54:28.129 INFO:teuthology.orchestra.run.smithi112.stdout:2449 writesame 0x9e94d13 thru 0x9e9d1ef data_size 0x1(0x84dd bytes) 2024-09-04T21:54:28.142 INFO:teuthology.orchestra.run.smithi112.stdout:2451 read 0x3e42e0b thru 0x3e490fe (0x62f4 bytes) 2024-09-04T21:54:28.143 INFO:teuthology.orchestra.run.smithi112.stdout:2452 read 0xd3ee0f2 thru 0xd3fcaf1 (0xea00 bytes) 2024-09-04T21:54:28.143 INFO:teuthology.orchestra.run.smithi112.stdout:2453 writesame 0x936ed57 thru 0x93730ff data_size 0x1(0x43a9 bytes) 2024-09-04T21:54:28.158 INFO:teuthology.orchestra.run.smithi112.stdout:2456 trunc from 0xdaa157d to 0xb32f5eb 2024-09-04T21:54:28.177 INFO:teuthology.orchestra.run.smithi112.stdout:2457 punch from 0x303c16e to 0x30492d2, (0xd164 bytes) 2024-09-04T21:54:28.177 INFO:teuthology.orchestra.run.smithi112.stdout:2458 writesame 0x72ae92e thru 0x72bdedc data_size 0x15(0xf5af bytes) 2024-09-04T21:54:28.186 INFO:teuthology.orchestra.run.smithi112.stdout:2459 writesame 0x84e8ef4 thru 0x84f8966 data_size 0x1(0xfa73 bytes) 2024-09-04T21:54:28.197 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:28.194+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:28.208 INFO:teuthology.orchestra.run.smithi112.stdout:2460 writesame 0xa64acb9 thru 0xa64f616 data_size 0x1(0x495e bytes) 2024-09-04T21:54:28.228 INFO:teuthology.orchestra.run.smithi112.stdout:2462 trunc from 0xb32f5eb to 0x76caedf 2024-09-04T21:54:28.246 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.246 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:28.260 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:28.280 INFO:teuthology.orchestra.run.smithi112.stdout:2463 write 0xe5a061b thru 0xe5af32b (0xed11 bytes) 2024-09-04T21:54:28.287 INFO:teuthology.orchestra.run.smithi112.stdout:2464 read 0x858f2ca thru 0x8596865 (0x759c bytes) 2024-09-04T21:54:28.287 INFO:teuthology.orchestra.run.smithi112.stdout:2465 read 0x527b16 thru 0x52800c (0x4f7 bytes) 2024-09-04T21:54:28.288 INFO:teuthology.orchestra.run.smithi112.stdout:2466 writesame 0x2691729 thru 0x2698c0a data_size 0x3(0x74e2 bytes) 2024-09-04T21:54:28.295 INFO:teuthology.orchestra.run.smithi112.stdout:2468 punch from 0x96ef671 to 0x96f4fa6, (0x5935 bytes) 2024-09-04T21:54:28.295 INFO:teuthology.orchestra.run.smithi112.stdout:2469 write 0x374b453 thru 0x3758f73 (0xdb21 bytes) 2024-09-04T21:54:28.298 INFO:teuthology.orchestra.run.smithi112.stdout:2471 read 0x3ec552d thru 0x3eceada (0x95ae bytes) 2024-09-04T21:54:28.298 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:28.294+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:28.298 INFO:teuthology.orchestra.run.smithi112.stdout:2472 write 0x846e263 thru 0x8479f72 (0xbd10 bytes) 2024-09-04T21:54:28.307 INFO:teuthology.orchestra.run.smithi112.stdout:2473 write 0x85a4aaa thru 0x85aa24c (0x57a3 bytes) 2024-09-04T21:54:28.312 INFO:teuthology.orchestra.run.smithi112.stdout:2474 punch from 0x8a3a76e to 0x8a48e2d, (0xe6bf bytes) 2024-09-04T21:54:28.312 INFO:teuthology.orchestra.run.smithi112.stdout:2476 write 0xf1c4c4 thru 0xf1e098 (0x1bd5 bytes) 2024-09-04T21:54:28.314 INFO:teuthology.orchestra.run.smithi112.stdout:2477 read 0x56f1685 thru 0x56fe4c6 (0xce42 bytes) 2024-09-04T21:54:28.314 INFO:teuthology.orchestra.run.smithi112.stdout:2479 write 0x66d456 thru 0x673a8c (0x6637 bytes) 2024-09-04T21:54:28.317 INFO:teuthology.orchestra.run.smithi112.stdout:2480 read 0xc586fbc thru 0xc596722 (0xf767 bytes) 2024-09-04T21:54:28.317 INFO:teuthology.orchestra.run.smithi112.stdout:2481 read 0xa83d859 thru 0xa849438 (0xbbe0 bytes) 2024-09-04T21:54:28.317 INFO:teuthology.orchestra.run.smithi112.stdout:2482 writesame 0x98ab328 thru 0x98b4c2f data_size 0x3b(0x9908 bytes) 2024-09-04T21:54:28.321 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.321 INFO:teuthology.orchestra.run.smithi112.stdout:2485 read 0xe151175 thru 0xe153709 (0x2595 bytes) 2024-09-04T21:54:28.321 INFO:teuthology.orchestra.run.smithi112.stdout:2487 read 0x6ee7042 thru 0x6ee8892 (0x1851 bytes) 2024-09-04T21:54:28.322 INFO:teuthology.orchestra.run.smithi112.stdout:2489 read 0xd2bd0c thru 0xd3bcc6 (0xffbb bytes) 2024-09-04T21:54:28.331 INFO:teuthology.orchestra.run.smithi112.stdout:2490 write 0xe5bb075 thru 0xe5bf1eb (0x4177 bytes) 2024-09-04T21:54:28.337 INFO:teuthology.orchestra.run.smithi112.stdout:2491 read 0x53403ec thru 0x534097e (0x593 bytes) 2024-09-04T21:54:28.338 INFO:teuthology.orchestra.run.smithi112.stdout:2492 read 0x604be2b thru 0x605ae09 (0xefdf bytes) 2024-09-04T21:54:28.353 INFO:teuthology.orchestra.run.smithi112.stdout:2493 trunc from 0xe5bf1ec to 0x557360c 2024-09-04T21:54:28.374 INFO:teuthology.orchestra.run.smithi112.stdout:2494 write 0x1e0aded thru 0x1e1a314 (0xf528 bytes) 2024-09-04T21:54:28.388 INFO:teuthology.orchestra.run.smithi112.stdout:2495 writesame 0x6d4be38 thru 0x6d4df21 data_size 0xb(0x20ea bytes) 2024-09-04T21:54:28.399 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:28.398+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:28.402 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.402 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:28.410 INFO:teuthology.orchestra.run.smithi112.stdout:2496 write 0xcd3ad73 thru 0xcd3de8c (0x311a bytes) 2024-09-04T21:54:28.415 INFO:teuthology.orchestra.run.smithi112.stdout:2497 read 0x2db5c63 thru 0x2dc1ae3 (0xbe81 bytes) 2024-09-04T21:54:28.416 INFO:teuthology.orchestra.run.smithi112.stdout:2502 writesame 0xa1ec9 thru 0xa3e1e data_size 0x15(0x1f56 bytes) 2024-09-04T21:54:28.418 INFO:teuthology.orchestra.run.smithi112.stdout:2503 read 0xada7919 thru 0xadb2a0a (0xb0f2 bytes) 2024-09-04T21:54:28.418 INFO:teuthology.orchestra.run.smithi112.stdout:2505 punch from 0xb4cd956 to 0xb4d6e1f, (0x94c9 bytes) 2024-09-04T21:54:28.418 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.418 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.418 INFO:teuthology.orchestra.run.smithi112.stdout:2509 read 0x3f4875d thru 0x3f50e0d (0x86b1 bytes) 2024-09-04T21:54:28.419 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:28.419 INFO:teuthology.orchestra.run.smithi112.stdout:2511 writesame 0x1978882 thru 0x1979422 data_size 0x1(0xba1 bytes) 2024-09-04T21:54:28.422 INFO:teuthology.orchestra.run.smithi112.stdout:2512 punch from 0xa1888d6 to 0xa1893da, (0xb04 bytes) 2024-09-04T21:54:28.422 INFO:teuthology.orchestra.run.smithi112.stdout:2514 writesame 0x3230386 thru 0x3230942 data_size 0x1(0x5bd bytes) 2024-09-04T21:54:28.424 INFO:teuthology.orchestra.run.smithi112.stdout:2515 punch from 0x4f3a39c to 0x4f44756, (0xa3ba bytes) 2024-09-04T21:54:28.424 INFO:teuthology.orchestra.run.smithi112.stdout:2517 write 0x48de9db thru 0x48ed2f2 (0xe918 bytes) 2024-09-04T21:54:28.428 INFO:teuthology.orchestra.run.smithi112.stdout:2519 write 0x1cb3d1d thru 0x1cbe4f0 (0xa7d4 bytes) 2024-09-04T21:54:28.430 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.430 INFO:teuthology.orchestra.run.smithi112.stdout:2525 read 0x354e882 thru 0x3552d0c (0x448b bytes) 2024-09-04T21:54:28.441 INFO:teuthology.orchestra.run.smithi112.stdout:2526 trunc from 0xcd3de8d to 0x6225133 2024-09-04T21:54:28.454 INFO:teuthology.orchestra.run.smithi112.stdout:2528 read 0x4268600 thru 0x4274b76 (0xc577 bytes) 2024-09-04T21:54:28.472 INFO:teuthology.orchestra.run.smithi112.stdout:2529 write 0xb817d58 thru 0xb818604 (0x8ad bytes) 2024-09-04T21:54:28.479 INFO:teuthology.orchestra.run.smithi112.stdout:2531 read 0x851527a thru 0x8518728 (0x34af bytes) 2024-09-04T21:54:28.479 INFO:teuthology.orchestra.run.smithi112.stdout:2532 read 0x33615e2 thru 0x3363873 (0x2292 bytes) 2024-09-04T21:54:28.480 INFO:teuthology.orchestra.run.smithi112.stdout:2534 write 0x1b8d8bf thru 0x1b99a1b (0xc15d bytes) 2024-09-04T21:54:28.483 INFO:teuthology.orchestra.run.smithi112.stdout:2535 write 0x2a0ecc2 thru 0x2a17380 (0x86bf bytes) 2024-09-04T21:54:28.486 INFO:teuthology.orchestra.run.smithi112.stdout:2536 punch from 0x3a961ab to 0x3a9ec87, (0x8adc bytes) 2024-09-04T21:54:28.486 INFO:teuthology.orchestra.run.smithi112.stdout:2537 write 0xfd5ad3 thru 0xfe0e07 (0xb335 bytes) 2024-09-04T21:54:28.490 INFO:teuthology.orchestra.run.smithi112.stdout:2539 read 0xad2f092 thru 0xad363e4 (0x7353 bytes) 2024-09-04T21:54:28.491 INFO:teuthology.orchestra.run.smithi112.stdout:2540 writesame 0x2deb5ea thru 0x2df70f5 data_size 0x2(0xbb0c bytes) 2024-09-04T21:54:28.500 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:28.498+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:28.501 INFO:teuthology.orchestra.run.smithi112.stdout:2542 read 0x9187b36 thru 0x918d060 (0x552b bytes) 2024-09-04T21:54:28.502 INFO:teuthology.orchestra.run.smithi112.stdout:2543 punch from 0x92b7b8 to 0x937547, (0xbd8f bytes) 2024-09-04T21:54:28.502 INFO:teuthology.orchestra.run.smithi112.stdout:2544 writesame 0x34921ca thru 0x349314b data_size 0xa(0xf82 bytes) 2024-09-04T21:54:28.513 INFO:teuthology.orchestra.run.smithi112.stdout:2545 trunc from 0xb818605 to 0x5be6114 2024-09-04T21:54:28.532 INFO:teuthology.orchestra.run.smithi112.stdout:2546 punch from 0x308bd49 to 0x309bc03, (0xfeba bytes) 2024-09-04T21:54:28.532 INFO:teuthology.orchestra.run.smithi112.stdout:2547 write 0x5857147 thru 0x5860728 (0x95e2 bytes) 2024-09-04T21:54:28.552 INFO:teuthology.orchestra.run.smithi112.stdout:2548 writesame 0xa49eaeb thru 0xa4a6f30 data_size 0x1(0x8446 bytes) 2024-09-04T21:54:28.552 INFO:teuthology.orchestra.run.smithi112.stdout:2549 punch from 0x5cb200d to 0x5cb6b6f, (0x4b62 bytes) 2024-09-04T21:54:28.552 INFO:teuthology.orchestra.run.smithi112.stdout:2550 read 0x1637b30 thru 0x1646231 (0xe702 bytes) 2024-09-04T21:54:28.554 INFO:teuthology.orchestra.run.smithi112.stdout:2552 write 0x1faf07e thru 0x1fb2503 (0x3486 bytes) 2024-09-04T21:54:28.555 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:54:28.557 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.557 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:28.560 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.560 INFO:teuthology.orchestra.run.smithi112.stdout:2554 punch from 0x4a44bac to 0x4a4c21b, (0x766f bytes) 2024-09-04T21:54:28.560 INFO:teuthology.orchestra.run.smithi112.stdout:2555 read 0x998ee1c thru 0x999b8c6 (0xcaab bytes) 2024-09-04T21:54:28.560 INFO:teuthology.orchestra.run.smithi112.stdout:2557 writesame 0xec927d thru 0xec9626 data_size 0x7(0x3aa bytes) 2024-09-04T21:54:28.564 INFO:teuthology.orchestra.run.smithi112.stdout:2558 read 0x9056119 thru 0x905ec35 (0x8b1d bytes) 2024-09-04T21:54:28.564 INFO:teuthology.orchestra.run.smithi112.stdout:2559 read 0x835e981 thru 0x836c041 (0xd6c1 bytes) 2024-09-04T21:54:28.565 INFO:teuthology.orchestra.run.smithi112.stdout:2560 trunc from 0xa4a6f31 to 0xa9fd048 2024-09-04T21:54:28.574 INFO:teuthology.orchestra.run.smithi112.stdout:2561 punch from 0x37168a0 to 0x3726240, (0xf9a0 bytes) 2024-09-04T21:54:28.574 INFO:teuthology.orchestra.run.smithi112.stdout:2562 read 0x6b9a985 thru 0x6ba0db8 (0x6434 bytes) 2024-09-04T21:54:28.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:28.585 INFO:teuthology.orchestra.run.smithi112.stdout:2565 trunc from 0xa9fd048 to 0x2793ce5 2024-09-04T21:54:28.608 INFO:teuthology.orchestra.run.smithi112.stdout:2567 read 0x151385c thru 0x151ec7e (0xb423 bytes) 2024-09-04T21:54:28.609 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:28.606+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:28.627 INFO:teuthology.orchestra.run.smithi112.stdout:2568 write 0x9d26798 thru 0x9d2f690 (0x8ef9 bytes) 2024-09-04T21:54:28.637 INFO:teuthology.orchestra.run.smithi112.stdout:2570 write 0x8db0a56 thru 0x8db8425 (0x79d0 bytes) 2024-09-04T21:54:28.643 INFO:teuthology.orchestra.run.smithi112.stdout:2571 write 0xf30440 thru 0xf3ce95 (0xca56 bytes) 2024-09-04T21:54:28.646 INFO:teuthology.orchestra.run.smithi112.stdout:2572 writesame 0x6c9277e thru 0x6c9aaa7 data_size 0x2(0x832a bytes) 2024-09-04T21:54:28.653 INFO:teuthology.orchestra.run.smithi112.stdout:2574 punch from 0xcdbe46 to 0xceaeab, (0xf065 bytes) 2024-09-04T21:54:28.653 INFO:teuthology.orchestra.run.smithi112.stdout:2575 read 0x7fd7ba thru 0x807a4e (0xa295 bytes) 2024-09-04T21:54:28.654 INFO:teuthology.orchestra.run.smithi112.stdout:2576 read 0x566f243 thru 0x567acbd (0xba7b bytes) 2024-09-04T21:54:28.654 INFO:teuthology.orchestra.run.smithi112.stdout:2578 read 0x38e4cc0 thru 0x38e931d (0x465e bytes) 2024-09-04T21:54:28.665 INFO:teuthology.orchestra.run.smithi112.stdout:2579 write 0xd0acc07 thru 0xd0afcbb (0x30b5 bytes) 2024-09-04T21:54:28.670 INFO:teuthology.orchestra.run.smithi112.stdout:2580 read 0x4cd0ea0 thru 0x4cdb374 (0xa4d5 bytes) 2024-09-04T21:54:28.670 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.670 INFO:teuthology.orchestra.run.smithi112.stdout:2582 read 0x38b54f1 thru 0x38c5004 (0xfb14 bytes) 2024-09-04T21:54:28.670 INFO:teuthology.orchestra.run.smithi112.stdout:2583 punch from 0x9d53076 to 0x9d5d1fd, (0xa187 bytes) 2024-09-04T21:54:28.670 INFO:teuthology.orchestra.run.smithi112.stdout:2584 punch from 0x34b5ab6 to 0x34c38cd, (0xde17 bytes) 2024-09-04T21:54:28.671 INFO:teuthology.orchestra.run.smithi112.stdout:2585 read 0xb264f09 thru 0xb273958 (0xea50 bytes) 2024-09-04T21:54:28.671 INFO:teuthology.orchestra.run.smithi112.stdout:2586 write 0xd04d5ee thru 0xd055478 (0x7e8b bytes) 2024-09-04T21:54:28.673 INFO:teuthology.orchestra.run.smithi112.stdout:2587 writesame 0x9b6bcdf thru 0x9b76807 data_size 0x1(0xab29 bytes) 2024-09-04T21:54:28.702 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:28.698+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:28.705 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.705 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:28.708 INFO:teuthology.orchestra.run.smithi112.stdout:2588 write 0x97f28d0 thru 0x97fb2ef (0x8a20 bytes) 2024-09-04T21:54:28.711 INFO:teuthology.orchestra.run.smithi112.stdout:2589 write 0xa9ba1c3 thru 0xa9c8ee4 (0xed22 bytes) 2024-09-04T21:54:28.721 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:28.722 INFO:teuthology.orchestra.run.smithi112.stdout:2590 write 0x43752e5 thru 0x43784bd (0x31d9 bytes) 2024-09-04T21:54:28.727 INFO:teuthology.orchestra.run.smithi112.stdout:2592 read 0xbc48674 thru 0xbc51a80 (0x940d bytes) 2024-09-04T21:54:28.728 INFO:teuthology.orchestra.run.smithi112.stdout:2593 punch from 0x6b10055 to 0x6b1e6c4, (0xe66f bytes) 2024-09-04T21:54:28.728 INFO:teuthology.orchestra.run.smithi112.stdout:2595 write 0xb5e43c0 thru 0xb5f36e5 (0xf326 bytes) 2024-09-04T21:54:28.735 INFO:teuthology.orchestra.run.smithi112.stdout:2596 read 0xb3e8d66 thru 0xb3f7ff6 (0xf291 bytes) 2024-09-04T21:54:28.739 INFO:teuthology.orchestra.run.smithi112.stdout:2597 trunc from 0xd0afcbc to 0xb091abb 2024-09-04T21:54:28.757 INFO:teuthology.orchestra.run.smithi112.stdout:2598 read 0xcc673e thru 0xcca105 (0x39c8 bytes) 2024-09-04T21:54:28.760 INFO:teuthology.orchestra.run.smithi112.stdout:2599 trunc from 0xb091abb to 0x9fb7ad8 2024-09-04T21:54:28.782 INFO:teuthology.orchestra.run.smithi112.stdout:2600 write 0x94440b6 thru 0x944b8f4 (0x783f bytes) 2024-09-04T21:54:28.786 INFO:teuthology.orchestra.run.smithi112.stdout:2603 writesame 0x454fac7 thru 0x4551887 data_size 0x1(0x1dc1 bytes) 2024-09-04T21:54:28.791 INFO:teuthology.orchestra.run.smithi112.stdout:2606 punch from 0x828439e to 0x828a9f6, (0x6658 bytes) 2024-09-04T21:54:28.798 INFO:teuthology.orchestra.run.smithi112.stdout:2607 trunc from 0x9fb7ad8 to 0xe5caf9e 2024-09-04T21:54:28.802 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:28.798+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:28.806 INFO:teuthology.orchestra.run.smithi112.stdout:2608 punch from 0x504ccec to 0x504f1f6, (0x250a bytes) 2024-09-04T21:54:28.806 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.806 INFO:teuthology.orchestra.run.smithi112.stdout:2613 punch from 0xcffe5de to 0xd00b17d, (0xcb9f bytes) 2024-09-04T21:54:28.806 INFO:teuthology.orchestra.run.smithi112.stdout:2616 write 0xa2bd7b0 thru 0xa2c33d5 (0x5c26 bytes) 2024-09-04T21:54:28.814 INFO:teuthology.orchestra.run.smithi112.stdout:2617 read 0x540485 thru 0x546898 (0x6414 bytes) 2024-09-04T21:54:28.815 INFO:teuthology.orchestra.run.smithi112.stdout:2618 read 0x831452 thru 0x83642b (0x4fda bytes) 2024-09-04T21:54:28.816 INFO:teuthology.orchestra.run.smithi112.stdout:2619 write 0x7618e2f thru 0x761f732 (0x6904 bytes) 2024-09-04T21:54:28.818 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.818 INFO:teuthology.orchestra.run.smithi112.stdout:2622 write 0x9064e1b thru 0x9072f2a (0xe110 bytes) 2024-09-04T21:54:28.823 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.823 INFO:teuthology.orchestra.run.smithi112.stdout:2625 writesame 0x42c036f thru 0x42c5e34 data_size 0x9(0x5ac6 bytes) 2024-09-04T21:54:28.829 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.829 INFO:teuthology.orchestra.run.smithi112.stdout:2628 read 0x5e9d5c thru 0x5f1ce5 (0x7f8a bytes) 2024-09-04T21:54:28.830 INFO:teuthology.orchestra.run.smithi112.stdout:2631 read 0x969b5f4 thru 0x969e4f3 (0x2f00 bytes) 2024-09-04T21:54:28.831 INFO:teuthology.orchestra.run.smithi112.stdout:2632 write 0xc2ee5a5 thru 0xc2ef2ab (0xd07 bytes) 2024-09-04T21:54:28.837 INFO:teuthology.orchestra.run.smithi112.stdout:2633 write 0xcc10995 thru 0xcc183e9 (0x7a55 bytes) 2024-09-04T21:54:28.844 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.850 INFO:teuthology.orchestra.run.smithi112.stdout:2635 trunc from 0xe5caf9e to 0xc345b25 2024-09-04T21:54:28.855 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.855 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:28.861 INFO:teuthology.orchestra.run.smithi112.stdout:2637 writesame 0x58b18d0 thru 0x58bb156 data_size 0x1(0x9887 bytes) 2024-09-04T21:54:28.871 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:28.875 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:28.879 INFO:teuthology.orchestra.run.smithi112.stdout:2639 trunc from 0xc345b25 to 0x94701ee 2024-09-04T21:54:28.895 INFO:teuthology.orchestra.run.smithi112.stdout:2641 read 0x5b1c268 thru 0x5b2551b (0x92b4 bytes) 2024-09-04T21:54:28.896 INFO:teuthology.orchestra.run.smithi112.stdout:2643 read 0x8d8ca81 thru 0x8d99c87 (0xd207 bytes) 2024-09-04T21:54:28.897 INFO:teuthology.orchestra.run.smithi112.stdout:2645 clone 13 order 21 su 8192 sc 2 2024-09-04T21:54:28.903 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:28.898+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:28.995 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:28.996 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:29.004 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:28.998+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:29.010 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:29.104 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:29.102+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:29.178 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:29.178 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:29.193 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:29.237 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:29.234+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:29.248 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:29.246+0000 7fb39a3ba7c0 -1 osd.7 21 log_to_monitors true 2024-09-04T21:54:29.306 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:29.306+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:29.360 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:29.360 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:29.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:29.420 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:29.418+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:29.497 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:29.497 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:29.509 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:29.506+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:29.511 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:29.591 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:29.590+0000 7fb38e195640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:54:29.610 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:29.606+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:29.689 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:29.690 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:29.704 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:29.706 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:29.754 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:29.750+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:29.800 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-09-04T21:54:29.810 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:29.806+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:29.875 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:29.875 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:29.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:29.911 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:29.912 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:29.913 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:29.910+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:29.930 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:29.931 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:29.948 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:29.948 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:29.951 INFO:teuthology.orchestra.run.smithi112.stdout:2649 write 0x20dc16d thru 0x20e3a54 (0x78e8 bytes) 2024-09-04T21:54:29.972 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:29.972 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:29.998 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:29.998 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:30.012 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:30.010+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:30.019 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:30.019 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:30.033 INFO:teuthology.orchestra.run.smithi112.stdout:2650 trunc from 0x94701ee to 0x39d55ad 2024-09-04T21:54:30.050 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:30.050 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:30.050 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:30.050 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:30.050 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:30.054 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:30.054 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:30.057 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-04T21:54:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:54:30.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:54:30.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:30.102 INFO:teuthology.orchestra.run.smithi112.stdout:2652 trunc from 0x39d55ad to 0x492e9a8 2024-09-04T21:54:30.109 INFO:teuthology.orchestra.run.smithi112.stdout:2653 read 0x24ab1fe thru 0x24b2328 (0x712b bytes) 2024-09-04T21:54:30.110 INFO:teuthology.orchestra.run.smithi112.stdout:2656 read 0x25fbdc7 thru 0x25fd8b3 (0x1aed bytes) 2024-09-04T21:54:30.111 INFO:teuthology.orchestra.run.smithi112.stdout:2657 read 0x13d9b96 thru 0x13e3a88 (0x9ef3 bytes) 2024-09-04T21:54:30.113 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:30.110+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:30.125 INFO:teuthology.orchestra.run.smithi112.stdout:2659 write 0x7dd2d40 thru 0x7dd4dfe (0x20bf bytes) 2024-09-04T21:54:30.143 INFO:teuthology.orchestra.run.smithi112.stdout:2661 writesame 0xa849ce8 thru 0xa850d8c data_size 0x1(0x70a5 bytes) 2024-09-04T21:54:30.143 INFO:teuthology.orchestra.run.smithi112.stdout:2662 punch from 0x793a0a3 to 0x79499ae, (0xf90b bytes) 2024-09-04T21:54:30.154 INFO:teuthology.orchestra.run.smithi112.stdout:2663 write 0xdc5214b thru 0xdc60a27 (0xe8dd bytes) 2024-09-04T21:54:30.161 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:30.161 INFO:teuthology.orchestra.run.smithi112.stdout:2665 write 0x4ce3bf thru 0x4d75c5 (0x9207 bytes) 2024-09-04T21:54:30.214 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:30.210+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7369 3.2 3:473dc813:::rbd_data.1188b1294ada.0000000000000008:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:30.229 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.112213004000001, 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035947, 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7369 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117103+0000", 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117106+0000", 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9799999999999998e-06 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117118+0000", 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1792e-05 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117149+0000", 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0412999999999998e-05 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117163+0000", 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4779e-05 2024-09-04T21:54:30.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117282+0000", 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011867 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.117441+0000", 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015941300000000001 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 5", 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.123847+0000", 2024-09-04T21:54:30.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064053850000000004 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152852+0000", 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.02900525 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152883+0000", 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0756999999999997e-05 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.152973+0000", 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0002999999999999e-05 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.153050+0000", 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7558000000000005e-05 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7714 3.7 3:e20eb1bd:::rbd_data.1188b1294ada.0000000000000030:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.397497946000001, 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034647877000000001, 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7714 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.236 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831818+0000", 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831822+0000", 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1599999999999998e-06 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831833+0000", 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1472e-05 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831865+0000", 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1707e-05 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.831880+0000", 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4871e-05 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.832016+0000", 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000136685 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.832254+0000", 2024-09-04T21:54:30.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023754099999999999 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866177+0000", 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.03392299 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866313+0000", 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135995 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.866466+0000", 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000153456 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:27327 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:e [read 4423680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:26.990920+0000", 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.238396324, 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031128517000000001, 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.241 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27327 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990920+0000", 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990920+0000", 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990921+0000", 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4300000000000001e-06 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990927+0000", 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6030000000000004e-06 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990944+0000", 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990958+0000", 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.021880+0000", 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.030922153000000001 2024-09-04T21:54:30.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.021953+0000", 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2721999999999999e-05 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022049+0000", 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5864000000000002e-05 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:27328 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:e [read 4554752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:26.990980+0000", 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.2383366549999999, 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031276348000000002, 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27328 2024-09-04T21:54:30.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990980+0000", 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990980+0000", 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990981+0000", 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.990986+0000", 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991004+0000", 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7969e-05 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991018+0000", 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3786e-05 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022095+0000", 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.03107644 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022148+0000", 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3735e-05 2024-09-04T21:54:30.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022256+0000", 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107688 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:27329 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:e [read 4685824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:26.991036+0000", 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.2382803949999999, 2024-09-04T21:54:30.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031434480000000001, 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27329 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991036+0000", 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991036+0000", 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991037+0000", 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1370000000000001e-06 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991067+0000", 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9731e-05 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991085+0000", 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8102999999999999e-05 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991100+0000", 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5311999999999998e-05 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022306+0000", 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031205521 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022373+0000", 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6790000000000003e-05 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022470+0000", 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7886000000000004e-05 2024-09-04T21:54:30.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:27330 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:e [read 4816896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:26.991123+0000", 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.2381934600000002, 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031536882000000002, 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27330 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991123+0000", 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991123+0000", 2024-09-04T21:54:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991125+0000", 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7060000000000001e-06 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991131+0000", 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0129999999999999e-06 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991148+0000", 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7509000000000001e-05 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991162+0000", 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3749e-05 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022518+0000", 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031356083999999999 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022572+0000", 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3814999999999999e-05 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022660+0000", 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8005999999999997e-05 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:27331 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:e [read 4947968~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:26.991181+0000", 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.2381350470000001, 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.03168091, 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27331 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991181+0000", 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991181+0000", 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991182+0000", 2024-09-04T21:54:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.13e-06 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991188+0000", 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991205+0000", 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7292e-05 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:26.991219+0000", 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3533000000000001e-05 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022694+0000", 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031475415999999999 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022761+0000", 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000001e-05 2024-09-04T21:54:30.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:27.022862+0000", 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010142 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31944 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1892352~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171892+0000", 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057424245999999998, 2024-09-04T21:54:30.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031504274999999998, 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31944 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171892+0000", 2024-09-04T21:54:30.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171892+0000", 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171892+0000", 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7300000000000002e-07 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171894+0000", 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171901+0000", 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909999999999998e-06 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171905+0000", 2024-09-04T21:54:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0629999999999999e-06 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203027+0000", 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031122329000000001 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203166+0000", 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013862999999999999 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203396+0000", 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023076699999999999 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31945 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1908736~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171909+0000", 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057407647999999999, 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031784845999999999, 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.269 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31945 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171909+0000", 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171909+0000", 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171909+0000", 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4900000000000002e-07 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171910+0000", 2024-09-04T21:54:30.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.265e-06 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171915+0000", 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5499999999999996e-06 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171917+0000", 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7350000000000001e-06 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203458+0000", 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031540213999999997 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203547+0000", 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9512999999999995e-05 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203694+0000", 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014631999999999999 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.273 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31946 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1925120~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171921+0000", 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057395235000000003, 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.032199084000000003, 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.274 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31946 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171921+0000", 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171921+0000", 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171922+0000", 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0999999999999999e-07 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171923+0000", 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.167e-06 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171927+0000", 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.33e-06 2024-09-04T21:54:30.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171930+0000", 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8569999999999999e-06 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203752+0000", 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.031821663 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.203893+0000", 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014108400000000001 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204120+0000", 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022757300000000001 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31947 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1941504~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171933+0000", 2024-09-04T21:54:30.278 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057382837999999999, 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.032488329000000003, 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31947 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171933+0000", 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171933+0000", 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171934+0000", 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.67e-07 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171935+0000", 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3740000000000001e-06 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171943+0000", 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6040000000000004e-06 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171947+0000", 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6630000000000001e-06 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204178+0000", 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.032230887999999999 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204270+0000", 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2115999999999997e-05 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204422+0000", 2024-09-04T21:54:30.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015141699999999999 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31948 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1957888~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171954+0000", 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057362135000000002, 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.032905416999999999, 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31948 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171954+0000", 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171954+0000", 2024-09-04T21:54:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171955+0000", 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5499999999999999e-07 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171956+0000", 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.28e-06 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171961+0000", 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7709999999999997e-06 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171963+0000", 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8959999999999999e-06 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204486+0000", 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.032522938000000001 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204627+0000", 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014061400000000001 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204860+0000", 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000232563 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.287 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31949 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1974272~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171968+0000", 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057348791000000003, 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.033192890000000003, 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31949 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171968+0000", 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171968+0000", 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171968+0000", 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2999999999999999e-07 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171969+0000", 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2640000000000001e-06 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171972+0000", 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4199999999999999e-06 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171975+0000", 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8310000000000001e-06 2024-09-04T21:54:30.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.204924+0000", 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.032948437999999997 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205013+0000", 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9587999999999997e-05 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205160+0000", 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000147119 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31950 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 1990656~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171979+0000", 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057337227999999997, 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.033596793, 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31950 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171979+0000", 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171979+0000", 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171979+0000", 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.22e-07 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171981+0000", 2024-09-04T21:54:30.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.229e-06 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171985+0000", 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3359999999999997e-06 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171988+0000", 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8940000000000002e-06 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205221+0000", 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.033233424999999997 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205356+0000", 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013493899999999999 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205576+0000", 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000219748 2024-09-04T21:54:30.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31951 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 2007040~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.171991+0000", 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057324976999999999, 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.033886649999999997, 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.297 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31951 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171991+0000", 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171991+0000", 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171992+0000", 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.36e-07 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171994+0000", 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.429e-06 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.171997+0000", 2024-09-04T21:54:30.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3579999999999999e-06 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172000+0000", 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7410000000000001e-06 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205633+0000", 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.033632566000000003 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205726+0000", 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3181000000000001e-05 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205878+0000", 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000152139 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.301 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31952 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 2023424~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.172004+0000", 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057312770999999998, 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034303898999999999, 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31952 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172004+0000", 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172004+0000", 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172004+0000", 2024-09-04T21:54:30.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3099999999999999e-07 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172005+0000", 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1319999999999999e-06 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172014+0000", 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8170000000000005e-06 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172017+0000", 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7280000000000001e-06 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.205939+0000", 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.033921382999999999 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206076+0000", 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000136873 2024-09-04T21:54:30.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206307+0000", 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000231735 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31953 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 2039808~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.172022+0000", 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057294673999999997, 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034591627, 2024-09-04T21:54:30.306 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31953 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172022+0000", 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172022+0000", 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172022+0000", 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7700000000000001e-07 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172023+0000", 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2419999999999999e-06 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172027+0000", 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.45e-06 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172029+0000", 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8660000000000002e-06 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206370+0000", 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034340347 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206460+0000", 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0008999999999998e-05 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206613+0000", 2024-09-04T21:54:30.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015343600000000001 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31954 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 2056192~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.172033+0000", 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057283309999999997, 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034988760000000001, 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31954 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172033+0000", 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172033+0000", 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172033+0000", 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1400000000000001e-07 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172034+0000", 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0899999999999999e-06 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172037+0000", 2024-09-04T21:54:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1690000000000001e-06 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172040+0000", 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7870000000000002e-06 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206671+0000", 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.034630299000000003 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.206807+0000", 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013628800000000001 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207022+0000", 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021491300000000001 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31955 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 2072576~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.172044+0000", 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057272735999999998, 2024-09-04T21:54:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035279043000000003, 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31955 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172044+0000", 2024-09-04T21:54:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172044+0000", 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172044+0000", 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1899999999999999e-07 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172045+0000", 2024-09-04T21:54:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9199999999999999e-07 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172049+0000", 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9339999999999999e-06 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172051+0000", 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7590000000000002e-06 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207082+0000", 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035030975999999998 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207174+0000", 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1617000000000001e-05 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207323+0000", 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148546 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:31956 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 2088960~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.172055+0000", 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.057261394, 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035670760000000003, 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 31956 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172055+0000", 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172055+0000", 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172055+0000", 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.36e-07 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172056+0000", 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.141e-06 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172063+0000", 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1500000000000002e-06 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.172068+0000", 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8099999999999997e-06 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207362+0000", 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.035293513999999998 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207499+0000", 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013684299999999999 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:30.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.207726+0000", 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000227066 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:30.324 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:30.326 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:30.326 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:30.322+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:30.354 INFO:teuthology.orchestra.run.smithi112.stdout:2666 trunc from 0xdc60a28 to 0xcc213ed 2024-09-04T21:54:30.374 INFO:teuthology.orchestra.run.smithi112.stdout:2667 read 0x8b3b168 thru 0x8b44655 (0x94ee bytes) 2024-09-04T21:54:30.374 INFO:teuthology.orchestra.run.smithi112.stdout:2668 read 0x8660739 thru 0x8663fa1 (0x3869 bytes) 2024-09-04T21:54:30.374 INFO:teuthology.orchestra.run.smithi112.stdout:2669 read 0x3a60ffa thru 0x3a66378 (0x537f bytes) 2024-09-04T21:54:30.375 INFO:teuthology.orchestra.run.smithi112.stdout:2670 punch from 0x4edc5d0 to 0x4eeb7a5, (0xf1d5 bytes) 2024-09-04T21:54:30.375 INFO:teuthology.orchestra.run.smithi112.stdout:2671 read 0x974f911 thru 0x975ae63 (0xb553 bytes) 2024-09-04T21:54:30.375 INFO:teuthology.orchestra.run.smithi112.stdout:2672 write 0xa2ea5e2 thru 0xa2f95aa (0xefc9 bytes) 2024-09-04T21:54:30.382 INFO:teuthology.orchestra.run.smithi112.stdout:2673 read 0x8d273bb thru 0x8d2772b (0x371 bytes) 2024-09-04T21:54:30.382 INFO:teuthology.orchestra.run.smithi112.stdout:2674 read 0x5bfb7f0 thru 0x5c02971 (0x7182 bytes) 2024-09-04T21:54:30.395 INFO:teuthology.orchestra.run.smithi112.stdout:2675 write 0xccfcb90 thru 0xcd025c6 (0x5a37 bytes) 2024-09-04T21:54:30.401 INFO:teuthology.orchestra.run.smithi112.stdout:2676 writesame 0xc441fcf thru 0xc44e5f6 data_size 0x44(0xc628 bytes) 2024-09-04T21:54:30.410 INFO:teuthology.orchestra.run.smithi112.stdout:2677 writesame 0x87faf24 thru 0x88092d7 data_size 0x26(0xe3b4 bytes) 2024-09-04T21:54:30.421 INFO:teuthology.orchestra.run.smithi112.stdout:2678 writesame 0x7bf866c thru 0x7c07615 data_size 0x1(0xefaa bytes) 2024-09-04T21:54:30.426 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:30.422+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:30.457 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:30.457 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:30.457 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:30.457 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:30.468 INFO:teuthology.orchestra.run.smithi112.stdout:2679 trunc from 0xcd025c7 to 0x7b2d4c5 2024-09-04T21:54:30.471 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:30.491 INFO:teuthology.orchestra.run.smithi112.stdout:2680 write 0x1a2edd6 thru 0x1a3c73b (0xd966 bytes) 2024-09-04T21:54:30.527 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:30.526+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:30.583 INFO:teuthology.orchestra.run.smithi112.stdout:2681 writesame 0x3381cab thru 0x33870ba data_size 0x28(0x5410 bytes) 2024-09-04T21:54:30.583 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:54:30.608 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:30.608 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:30.608 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:30.608 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:30.608 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:30.623 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:30.666 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:30.666+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:30.729 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:30.726+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:30.787 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:30.787 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:30.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:30.787 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:30.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31873 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1777664~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224240+0000", 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56113032200000001, 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098495071000000003, 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31873 2024-09-04T21:54:30.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224240+0000", 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224240+0000", 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224241+0000", 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5600000000000001e-06 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224246+0000", 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2429999999999996e-06 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224263+0000", 2024-09-04T21:54:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6765999999999998e-05 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224276+0000", 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3213e-05 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.322481+0000", 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098204231000000003 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.322574+0000", 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3417000000000004e-05 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.322735+0000", 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000160641 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.792 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31874 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1794048~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224295+0000", 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56107477699999997, 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098760440000000005, 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31874 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224295+0000", 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224295+0000", 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224296+0000", 2024-09-04T21:54:30.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8299999999999995e-07 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224302+0000", 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8830000000000003e-06 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224319+0000", 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6969999999999998e-05 2024-09-04T21:54:30.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224332+0000", 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2869e-05 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.322798+0000", 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098466215999999995 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.322895+0000", 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7458000000000001e-05 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323056+0000", 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000160061 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.797 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31875 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1810432~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224350+0000", 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56102030899999999, 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099021801000000007, 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31875 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224350+0000", 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224350+0000", 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224351+0000", 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0330000000000001e-06 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224356+0000", 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9899999999999997e-06 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224390+0000", 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4218999999999999e-05 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224404+0000", 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3959e-05 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323119+0000", 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098714912000000002 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323212+0000", 2024-09-04T21:54:30.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3442000000000004e-05 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323371+0000", 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015924600000000001 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31876 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1826816~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224423+0000", 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56094670800000002, 2024-09-04T21:54:30.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099293737000000007, 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31876 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224423+0000", 2024-09-04T21:54:30.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224423+0000", 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224425+0000", 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.39e-06 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224441+0000", 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6002999999999999e-05 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224459+0000", 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8059000000000001e-05 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224472+0000", 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3492999999999999e-05 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323449+0000", 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098977022999999997 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323544+0000", 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5236999999999996e-05 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323717+0000", 2024-09-04T21:54:30.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017253200000000001 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31877 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1843200~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224492+0000", 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56087836400000002, 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099554725999999996, 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.807 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31877 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224492+0000", 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224492+0000", 2024-09-04T21:54:30.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224493+0000", 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.153e-06 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224498+0000", 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3140000000000003e-06 2024-09-04T21:54:30.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224515+0000", 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6795000000000001e-05 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224533+0000", 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7807000000000001e-05 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323783+0000", 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099250222999999999 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.323882+0000", 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9549000000000005e-05 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324046+0000", 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016388500000000001 2024-09-04T21:54:30.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31878 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1859584~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224551+0000", 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56081935599999999, 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099821233999999995, 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31878 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224551+0000", 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224551+0000", 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224552+0000", 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224557+0000", 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.31e-06 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224573+0000", 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6133e-05 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224586+0000", 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3206e-05 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324112+0000", 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099525930999999998 2024-09-04T21:54:30.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324208+0000", 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5987000000000001e-05 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324372+0000", 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016352399999999999 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31879 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1875968~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224608+0000", 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56076153500000003, 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10008574100000001, 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.817 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31879 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224608+0000", 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224608+0000", 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224610+0000", 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.207e-06 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224615+0000", 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.733e-06 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224633+0000", 2024-09-04T21:54:30.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7603000000000001e-05 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224646+0000", 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3523000000000001e-05 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324437+0000", 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099790262000000005 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324532+0000", 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4887000000000001e-05 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324694+0000", 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162526 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31880 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1892352~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224664+0000", 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56070566300000002, 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100348044, 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31880 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.822 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224664+0000", 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224664+0000", 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224665+0000", 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1599999999999999e-06 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224671+0000", 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7300000000000002e-06 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224688+0000", 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7351999999999999e-05 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224703+0000", 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.432e-05 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324759+0000", 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100055937 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.324852+0000", 2024-09-04T21:54:30.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3772999999999998e-05 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325012+0000", 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000159772 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31881 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1908736~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224721+0000", 2024-09-04T21:54:30.826 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56064927899999994, 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100613381, 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31881 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224721+0000", 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224721+0000", 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224722+0000", 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1170000000000001e-06 2024-09-04T21:54:30.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224727+0000", 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1329999999999998e-06 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224744+0000", 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6969e-05 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224758+0000", 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3784000000000001e-05 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325076+0000", 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100318697 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325173+0000", 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6818999999999996e-05 2024-09-04T21:54:30.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325334+0000", 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016086200000000001 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31882 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1925120~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224775+0000", 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56059499700000004, 2024-09-04T21:54:30.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10087771800000001, 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31882 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224775+0000", 2024-09-04T21:54:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224775+0000", 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224776+0000", 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4160000000000001e-06 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224781+0000", 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.814e-06 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224799+0000", 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7486000000000002e-05 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224812+0000", 2024-09-04T21:54:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3553000000000001e-05 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325398+0000", 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100585805 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325491+0000", 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2699000000000001e-05 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325653+0000", 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016194500000000001 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31883 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1941504~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224830+0000", 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56054023600000003, 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101145099, 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31883 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224830+0000", 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224830+0000", 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224831+0000", 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-06 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224837+0000", 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4870000000000002e-06 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224853+0000", 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6549e-05 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224867+0000", 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.378e-05 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325717+0000", 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100849808 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325808+0000", 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1760000000000002e-05 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.325975+0000", 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000166322 2024-09-04T21:54:30.840 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31884 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1957888~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224907+0000", 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.560462558, 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101393149, 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31884 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224907+0000", 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224907+0000", 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224908+0000", 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224913+0000", 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7060000000000003e-06 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224924+0000", 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0873000000000001e-05 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224932+0000", 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7770000000000002e-06 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326042+0000", 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10110986700000001 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326137+0000", 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5055000000000004e-05 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326300+0000", 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016389099999999999 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31885 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1974272~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224946+0000", 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.560424323, 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101666649, 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31885 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.846 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224946+0000", 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224946+0000", 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224947+0000", 2024-09-04T21:54:30.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5300000000000002e-07 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224950+0000", 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4000000000000001e-06 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224959+0000", 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1230000000000006e-06 2024-09-04T21:54:30.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224968+0000", 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326363+0000", 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101394875 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326455+0000", 2024-09-04T21:54:30.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2478999999999993e-05 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326612+0000", 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015710600000000001 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31886 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1990656~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.224978+0000", 2024-09-04T21:54:30.850 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56039229099999999, 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101955965, 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31886 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224978+0000", 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224978+0000", 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224978+0000", 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7299999999999995e-07 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224981+0000", 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0139999999999999e-06 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224990+0000", 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5229999999999995e-06 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.224998+0000", 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7430000000000002e-06 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326676+0000", 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101678883 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326771+0000", 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5052999999999999e-05 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326934+0000", 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016207600000000001 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31887 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 2007040~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.225007+0000", 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56036307900000004, 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102244927, 2024-09-04T21:54:30.855 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31887 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225007+0000", 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225007+0000", 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225007+0000", 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225011+0000", 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.083e-06 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225019+0000", 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6500000000000002e-06 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225026+0000", 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.326999+0000", 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101972411 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327093+0000", 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3963999999999997e-05 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327252+0000", 2024-09-04T21:54:30.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015917199999999999 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31888 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 2023424~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.225035+0000", 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56033486300000002, 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102554486, 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.860 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31888 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225035+0000", 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225035+0000", 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225036+0000", 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3e-07 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225048+0000", 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.186e-05 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225057+0000", 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7829999999999993e-06 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225065+0000", 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3089999999999998e-06 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327315+0000", 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102250444 2024-09-04T21:54:30.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327405+0000", 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9682e-05 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327589+0000", 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000184778 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31889 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 2039808~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.225075+0000", 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56029530100000002, 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102850046, 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31889 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.865 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225075+0000", 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225075+0000", 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225075+0000", 2024-09-04T21:54:30.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5400000000000001e-07 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225078+0000", 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1099999999999999e-06 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225087+0000", 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6889999999999993e-06 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225094+0000", 2024-09-04T21:54:30.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0060000000000003e-06 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327655+0000", 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102560473 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327756+0000", 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000101687 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327925+0000", 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016842699999999999 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31890 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 2056192~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.225103+0000", 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56026682500000002, 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103141766, 2024-09-04T21:54:30.869 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31890 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225103+0000", 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225103+0000", 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225104+0000", 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3e-07 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225107+0000", 2024-09-04T21:54:30.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0429999999999999e-06 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225115+0000", 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5529999999999993e-06 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225122+0000", 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0400000000000004e-06 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.327991+0000", 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102868531 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328083+0000", 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1852000000000001e-05 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328245+0000", 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016211700000000001 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.873 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31891 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 2072576~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.225131+0000", 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56023855899999997, 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103437932, 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.874 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31891 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225131+0000", 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225131+0000", 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225132+0000", 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3600000000000003e-07 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225135+0000", 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8770000000000002e-06 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225144+0000", 2024-09-04T21:54:30.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7229999999999993e-06 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225151+0000", 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3529999999999997e-06 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328315+0000", 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103163846 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328407+0000", 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1906999999999999e-05 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328569+0000", 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016259 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.878 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:31892 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 2088960~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.225160+0000", 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56021007, 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103725128, 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 31892 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:30.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225160+0000", 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225160+0000", 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225160+0000", 2024-09-04T21:54:30.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1600000000000001e-07 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225163+0000", 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0199999999999999e-06 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225172+0000", 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6000000000000007e-06 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:30.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.225179+0000", 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.973e-06 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328633+0000", 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103453883 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328727+0000", 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3682000000000002e-05 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:30.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.328885+0000", 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015835400000000001 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:30.883 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:30.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:30.885 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:30.882+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:30.986 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:30.982+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:31.012 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:31.012 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:31.013 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:31.013 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:31.027 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:31.086 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:31.086+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:31.145 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:31.145 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:31.146 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:31.146 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:31.146 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:31.159 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:31.207 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:31.206+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:31.278 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19512 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 983040~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636624+0000", 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640323161, 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025024366999999999, 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19512 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636624+0000", 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636624+0000", 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636625+0000", 2024-09-04T21:54:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636652+0000", 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7061000000000001e-05 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636663+0000", 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1080000000000001e-05 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636671+0000", 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.503e-06 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661350+0000", 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024679570000000001 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661470+0000", 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119564 2024-09-04T21:54:31.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661649+0000", 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000178889 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19513 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1146880~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636704+0000", 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640243835, 2024-09-04T21:54:31.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025639926, 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19513 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636704+0000", 2024-09-04T21:54:31.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636704+0000", 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636705+0000", 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636709+0000", 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8759999999999998e-06 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636724+0000", 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5432999999999999e-05 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636731+0000", 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661777+0000", 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025045290000000001 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662043+0000", 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000265868 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662344+0000", 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030111599999999998 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.288 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19514 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1310720~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636772+0000", 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640176050999999, 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025968494000000002, 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19514 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636772+0000", 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636772+0000", 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636773+0000", 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1030000000000001e-06 2024-09-04T21:54:31.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636776+0000", 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7529999999999999e-06 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636800+0000", 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3728999999999999e-05 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636808+0000", 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9859999999999994e-06 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662449+0000", 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025640616000000001 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662580+0000", 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013143399999999999 2024-09-04T21:54:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662740+0000", 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015987299999999999 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19515 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1556480~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636821+0000", 2024-09-04T21:54:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640126647000001, 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026552834000000001, 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19515 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636821+0000", 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636821+0000", 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636822+0000", 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7000000000000003e-07 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636825+0000", 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4960000000000001e-06 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636835+0000", 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2930000000000006e-06 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636842+0000", 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0169999999999999e-06 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662841+0000", 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025999586000000002 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663075+0000", 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000233289 2024-09-04T21:54:31.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663374+0000", 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029918300000000001 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19516 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1720320~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636852+0000", 2024-09-04T21:54:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640095599, 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027108402, 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19516 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636852+0000", 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636852+0000", 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636853+0000", 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3699999999999999e-07 2024-09-04T21:54:31.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636856+0000", 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1099999999999999e-06 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636865+0000", 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636872+0000", 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8859999999999998e-06 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663502+0000", 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026630233999999999 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663654+0000", 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151886 2024-09-04T21:54:31.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663960+0000", 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030664899999999998 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19517 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1884160~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636881+0000", 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640066302999999, 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027694871999999999, 2024-09-04T21:54:31.303 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19517 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636881+0000", 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636881+0000", 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636882+0000", 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636885+0000", 2024-09-04T21:54:31.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9239999999999999e-06 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636893+0000", 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6719999999999993e-06 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636901+0000", 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1500000000000002e-06 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664058+0000", 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027157642999999999 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664285+0000", 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022626600000000001 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664576+0000", 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029158400000000001 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.307 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19518 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2293760~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636947+0000", 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.640000923000001, 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028244814, 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.308 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19518 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636947+0000", 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636947+0000", 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636948+0000", 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0839999999999999e-06 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636952+0000", 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9829999999999998e-06 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636963+0000", 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1715999999999999e-05 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636971+0000", 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664671+0000", 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027699745000000001 2024-09-04T21:54:31.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664913+0000", 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024215599999999999 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665191+0000", 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027864699999999998 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.312 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19519 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2457600~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639965905, 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028816913999999999, 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.313 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19519 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636986+0000", 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636994+0000", 2024-09-04T21:54:31.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8599999999999999e-06 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637002+0000", 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1489999999999997e-06 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665282+0000", 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028280784999999999 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665515+0000", 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232449 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665799+0000", 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028375699999999999 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19520 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2621440~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637079+0000", 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639868712, 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029314778, 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19520 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.318 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637079+0000", 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637079+0000", 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637080+0000", 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.097e-06 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637083+0000", 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4429999999999999e-06 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637093+0000", 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8789999999999998e-06 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637100+0000", 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1330000000000001e-06 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665890+0000", 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028789848999999999 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666123+0000", 2024-09-04T21:54:31.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232642 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666394+0000", 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027073499999999998 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19521 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2867200~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637111+0000", 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639836603999999, 2024-09-04T21:54:31.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029880898999999999, 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19521 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637111+0000", 2024-09-04T21:54:31.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637111+0000", 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637112+0000", 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9699999999999995e-07 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637115+0000", 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3629999999999998e-06 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637124+0000", 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8769999999999999e-06 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637131+0000", 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0230000000000004e-06 2024-09-04T21:54:31.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666478+0000", 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029346819 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666717+0000", 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023884899999999999 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666992+0000", 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000275271 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19522 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3031040~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637175+0000", 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639772633, 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030447393, 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.327 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19522 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637175+0000", 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637175+0000", 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637176+0000", 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9300000000000006e-07 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637180+0000", 2024-09-04T21:54:31.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7629999999999998e-06 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637190+0000", 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9469999999999998e-06 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637199+0000", 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9490000000000002e-06 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667082+0000", 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029883581999999999 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667331+0000", 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024896800000000002 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667622+0000", 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000291191 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.331 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19523 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3194880~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637210+0000", 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639737818, 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030852570999999999, 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19523 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637210+0000", 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637210+0000", 2024-09-04T21:54:31.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637211+0000", 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-06 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637214+0000", 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.366e-06 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637223+0000", 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8429999999999999e-06 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637230+0000", 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0330000000000003e-06 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667728+0000", 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030497570000000002 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667887+0000", 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015933700000000001 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668062+0000", 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017534500000000001 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19524 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3604480~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637252+0000", 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639695567, 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.031460546999999998, 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19524 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637252+0000", 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637252+0000", 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637253+0000", 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637278+0000", 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5632000000000001e-05 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637290+0000", 2024-09-04T21:54:31.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2055999999999999e-05 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637298+0000", 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9799999999999998e-06 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668168+0000", 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030870056 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668431+0000", 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026247800000000002 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668713+0000", 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028162100000000002 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.341 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19525 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3768320~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637346+0000", 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639601321000001, 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.031987768999999999, 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19525 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.342 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637346+0000", 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637346+0000", 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637347+0000", 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637351+0000", 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1699999999999999e-06 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637362+0000", 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0363000000000001e-05 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637369+0000", 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2799999999999998e-06 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668802+0000", 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.031433097 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669051+0000", 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024841199999999999 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669334+0000", 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028349700000000001 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19526 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3932160~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.346 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637380+0000", 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639567318999999, 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032563683000000003, 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19526 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637380+0000", 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637380+0000", 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637381+0000", 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0759999999999999e-06 2024-09-04T21:54:31.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637384+0000", 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.174e-06 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637393+0000", 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637400+0000", 2024-09-04T21:54:31.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0260000000000002e-06 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669425+0000", 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032024997999999999 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669670+0000", 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024425500000000003 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669944+0000", 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000274338 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19527 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 4177920~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637410+0000", 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639537456999999, 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032943358999999998, 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:31.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19527 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637410+0000", 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637410+0000", 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637411+0000", 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637414+0000", 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0460000000000001e-06 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637422+0000", 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6160000000000002e-06 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637429+0000", 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0870000000000001e-06 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670050+0000", 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032620709999999997 2024-09-04T21:54:31.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670186+0000", 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135921 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670353+0000", 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016733899999999999 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19541 3.c 3:35e97e9b:::rbd_data.118892c64562.0000000000000033:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 950272] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637439+0000", 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.639508352, 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.038452027, 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.356 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19541 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637439+0000", 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637439+0000", 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637440+0000", 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4700000000000001e-07 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637684+0000", 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024439200000000001 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637701+0000", 2024-09-04T21:54:31.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6356000000000001e-05 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637711+0000", 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0219e-05 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670035+0000", 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032324031000000003 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670393+0000", 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035815700000000001 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.671203+0000", 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00080962700000000005 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675556+0000", 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043532600000000003 2024-09-04T21:54:31.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675594+0000", 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7509000000000003e-05 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675776+0000", 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000182981 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675891+0000", 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011474800000000001 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19551 3.1c 3:3bdbea5f:::rbd_data.118892c64562.0000000000000038:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 950272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.646954+0000", 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.629993886999999, 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027736934000000001, 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.362 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19551 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646954+0000", 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646954+0000", 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646956+0000", 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0140000000000001e-06 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646966+0000", 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9420000000000007e-06 2024-09-04T21:54:31.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646991+0000", 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4986000000000001e-05 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.647005+0000", 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4036000000000001e-05 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670401+0000", 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023396107999999999 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670580+0000", 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017891200000000001 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670852+0000", 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027222200000000002 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674314+0000", 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034617480000000002 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674348+0000", 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4668000000000001e-05 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674568+0000", 2024-09-04T21:54:31.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021935699999999999 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674691+0000", 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000122941 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:54:31.368 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.8351822049999997, 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:54:31.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.374 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.6506411500000002, 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:54:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:54:31.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:54:31.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.381 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.381 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.381 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:31.382 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:31.383 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:31.378+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:31.483 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:31.482+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:31.552 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:31.552 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:31.552 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:31.553 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:31.567 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:31.578 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:54:31.583 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:31.578+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:31.684 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:31.682+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:31.748 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:31.749 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:31.763 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:31.810 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:31.810+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1045 3.10 3:08ed8dc3:::rbd_data.11882e693cd0.000000000000002f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.970075926, 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054299810000000004, 2024-09-04T21:54:31.887 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1045 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912477+0000", 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8369999999999999e-06 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912492+0000", 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.428e-05 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912529+0000", 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7673000000000003e-05 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912547+0000", 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7689e-05 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912710+0000", 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162526 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912993+0000", 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028371199999999998 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913324+0000", 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033046099999999999 2024-09-04T21:54:31.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915482+0000", 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021584989999999999 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917665+0000", 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021827449999999998 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917778+0000", 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000113063 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917904+0000", 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012549600000000001 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1101 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:31.893 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.462219884, 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0067943780000000002, 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1101 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420333+0000", 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6169999999999998e-06 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420347+0000", 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-05 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420381+0000", 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4122999999999998e-05 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420398+0000", 2024-09-04T21:54:31.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6286999999999998e-05 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420568+0000", 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017088199999999999 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420589+0000", 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0513000000000002e-05 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424142+0000", 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035527359999999999 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424352+0000", 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021066300000000001 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424715+0000", 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000362233 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426277+0000", 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015626069999999999 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426969+0000", 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069220800000000004 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427060+0000", 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1057000000000006e-05 2024-09-04T21:54:31.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427124+0000", 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3522000000000006e-05 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7519 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 24.387520524999999, 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054280980000000001, 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.900 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7519 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495033+0000", 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7299999999999999e-06 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495047+0000", 2024-09-04T21:54:31.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4066000000000001e-05 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495089+0000", 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2700000000000001e-05 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495112+0000", 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2965000000000001e-05 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495230+0000", 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117581 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495238+0000", 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.388e-06 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498788+0000", 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035494939999999998 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498872+0000", 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4190999999999994e-05 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498983+0000", 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110739 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.499844+0000", 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086077500000000004 2024-09-04T21:54:31.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500358+0000", 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051428599999999997 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500414+0000", 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5689999999999997e-05 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500457+0000", 2024-09-04T21:54:31.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3492999999999998e-05 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7675 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1179648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 24.146729110999999, 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057294420000000004, 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.907 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7675 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735829+0000", 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8079999999999994e-06 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736259+0000", 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042982200000000001 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736276+0000", 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7124000000000001e-05 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736287+0000", 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0899e-05 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736380+0000", 2024-09-04T21:54:31.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3304e-05 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736530+0000", 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014963300000000001 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736871+0000", 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034072099999999999 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739225+0000", 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023541030000000002 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741419+0000", 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021947189999999999 2024-09-04T21:54:31.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741495+0000", 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5087999999999997e-05 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741550+0000", 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5220999999999999e-05 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7875 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.303408224000002, 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052294439999999998, 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7875 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.914 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579142+0000", 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0500000000000002e-07 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579147+0000", 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1120000000000004e-06 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579154+0000", 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.097e-06 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579159+0000", 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4870000000000004e-06 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581460+0000", 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023010230000000001 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581541+0000", 2024-09-04T21:54:31.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0939999999999994e-05 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581739+0000", 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019828000000000001 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584176+0000", 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002437051 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584259+0000", 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2831999999999996e-05 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584320+0000", 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1204000000000006e-05 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584371+0000", 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0612999999999998e-05 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14753 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.088543+0000", 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.794006275999999, 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056008339999999998, 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.920 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14753 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088543+0000", 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088543+0000", 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088545+0000", 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.73e-06 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088553+0000", 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4230000000000005e-06 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088574+0000", 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0749000000000001e-05 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088584+0000", 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.027e-05 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.090476+0000", 2024-09-04T21:54:31.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018915679999999999 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.090753+0000", 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000276913 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.091809+0000", 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001056162 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.093605+0000", 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017957019999999999 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.093944+0000", 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033892699999999999 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.094033+0000", 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9538999999999998e-05 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.094144+0000", 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110851 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14768 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.160622+0000", 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.721927714, 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0069038479999999998, 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.926 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14768 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160622+0000", 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160622+0000", 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160625+0000", 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4070000000000002e-06 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160641+0000", 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5536000000000002e-05 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160671+0000", 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0660999999999997e-05 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160693+0000", 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1716e-05 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160837+0000", 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014385999999999999 2024-09-04T21:54:31.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161062+0000", 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022511499999999999 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.162272+0000", 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001209578 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167334+0000", 2024-09-04T21:54:31.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050621470000000003 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167356+0000", 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2351999999999999e-05 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167455+0000", 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8392000000000003e-05 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167526+0000", 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1083999999999999e-05 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14769 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.721774278, 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.008898853, 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.932 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14769 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160777+0000", 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.486e-06 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160786+0000", 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3740000000000004e-06 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160806+0000", 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0165000000000001e-05 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160822+0000", 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6203000000000001e-05 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163870+0000", 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003047812 2024-09-04T21:54:31.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.164137+0000", 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000266643 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.164872+0000", 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073519099999999995 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167623+0000", 2024-09-04T21:54:31.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027509129999999998 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169518+0000", 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001895274 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169593+0000", 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4270000000000006e-05 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169674+0000", 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1521999999999996e-05 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14770 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.721509536999999, 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0090832199999999995, 2024-09-04T21:54:31.938 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14770 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161042+0000", 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4930000000000001e-06 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161054+0000", 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1466e-05 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161079+0000", 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5114999999999999e-05 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161097+0000", 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8315999999999999e-05 2024-09-04T21:54:31.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.165726+0000", 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0046287849999999998 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.165869+0000", 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014293300000000001 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.166506+0000", 2024-09-04T21:54:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063653199999999998 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169697+0000", 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031909410000000001 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169867+0000", 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170478 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170053+0000", 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018593099999999999 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170123+0000", 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0229999999999997e-05 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14771 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.719216829000001, 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089354540000000007, 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14771 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163335+0000", 2024-09-04T21:54:31.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7030000000000002e-06 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163350+0000", 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4523e-05 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163387+0000", 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6817999999999999e-05 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163408+0000", 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1715000000000001e-05 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167677+0000", 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042681560000000004 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167880+0000", 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020378 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.168602+0000", 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00072149099999999999 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170348+0000", 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017465849999999999 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172076+0000", 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001727836 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172205+0000", 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012882700000000001 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172268+0000", 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3020000000000003e-05 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.950 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:23212 3.0 e41/38 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head v 41'114, mlcod=39'112)", 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.564191360000001, 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0092641379999999999, 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318361+0000", 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:31.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322409+0000", 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040474400000000002 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322451+0000", 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2240999999999998e-05 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322473+0000", 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1676e-05 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322671+0000", 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019802099999999999 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322713+0000", 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2645000000000003e-05 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.327478+0000", 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047643989999999999 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.327622+0000", 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014443 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.955 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:23214 3.1e e41/22 3:7a40d9fa:::rbd_data.118892c64562.0000000000000009:head v 41'75, mlcod=41'73)", 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.549368064999999, 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088767499999999992, 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333184+0000", 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9469999999999999e-06 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335653+0000", 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024681690000000001 2024-09-04T21:54:31.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335684+0000", 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1112000000000003e-05 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335697+0000", 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3329999999999999e-05 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335861+0000", 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016441900000000001 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335929+0000", 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7953999999999993e-05 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.341970+0000", 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006040803 2024-09-04T21:54:31.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.342058+0000", 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8016000000000006e-05 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:23926 3.a 3:547e81fb:::rbd_data.118892c64562.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1622016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.134393009, 2024-09-04T21:54:31.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010368176, 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 23926 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748159+0000", 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8729999999999999e-06 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750340+0000", 2024-09-04T21:54:31.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021802879999999998 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750372+0000", 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2758e-05 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750395+0000", 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2368999999999999e-05 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750558+0000", 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016326000000000001 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750850+0000", 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000292117 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.751390+0000", 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054000999999999995 2024-09-04T21:54:31.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.757506+0000", 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061158460000000003 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758289+0000", 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078254599999999996 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758407+0000", 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011886400000000001 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758525+0000", 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011724500000000001 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24315 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:31.966 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2691827, 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074709240000000003, 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24315 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:31.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613370+0000", 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.49e-06 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613385+0000", 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5112e-05 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613421+0000", 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5246000000000001e-05 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613438+0000", 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7051999999999999e-05 2024-09-04T21:54:31.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613585+0000", 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014730799999999999 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613612+0000", 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7212e-05 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.617673+0000", 2024-09-04T21:54:31.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040609959999999999 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.617898+0000", 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022498400000000001 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.618232+0000", 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033338999999999998 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.619944+0000", 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001712746 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620686+0000", 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074206200000000004 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620774+0000", 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7249000000000004e-05 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620838+0000", 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4077000000000003e-05 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24372 3.a 3:547e81fb:::rbd_data.118892c64562.0000000000000002:head [stat,truncate 1081344] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2001553529999995, 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088444000000000005, 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.973 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24372 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682398+0000", 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.924e-06 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682412+0000", 2024-09-04T21:54:31.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3782e-05 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682458+0000", 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5738000000000002e-05 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682483+0000", 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5313e-05 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682668+0000", 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018540200000000001 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682935+0000", 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000266412 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.683466+0000", 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053094700000000004 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.689562+0000", 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060959769999999998 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.690972+0000", 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014100499999999999 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691102+0000", 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013006399999999999 2024-09-04T21:54:31.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691239+0000", 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013679100000000001 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:24378 3.0 e42/38 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head v 42'122, mlcod=41'119)", 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.685916+0000", 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1966336829999999, 2024-09-04T21:54:31.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052256239999999999, 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685916+0000", 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685916+0000", 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685918+0000", 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700000000000001e-06 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685935+0000", 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7169000000000001e-05 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685966+0000", 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0889000000000002e-05 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685985+0000", 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8258999999999999e-05 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.686215+0000", 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023046699999999999 2024-09-04T21:54:31.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.686268+0000", 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3424999999999998e-05 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691024+0000", 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047552200000000001 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691141+0000", 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117725 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24689 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.075516+0000", 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8070335649999993, 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052406229999999998, 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24689 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075516+0000", 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075516+0000", 2024-09-04T21:54:31.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075517+0000", 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3430000000000001e-06 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075522+0000", 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0950000000000003e-06 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075535+0000", 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2975e-05 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075541+0000", 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4700000000000001e-06 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075608+0000", 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7175000000000007e-05 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075700+0000", 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2453000000000004e-05 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076089+0000", 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000388496 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080300+0000", 2024-09-04T21:54:31.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042111060000000001 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080487+0000", 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018713700000000001 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080642+0000", 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015494400000000001 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080757+0000", 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114429 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24690 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.075702+0000", 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8068474210000005, 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006662482, 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.990 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24690 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075702+0000", 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075702+0000", 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075703+0000", 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075712+0000", 2024-09-04T21:54:31.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2269999999999999e-06 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075725+0000", 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2262000000000001e-05 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075732+0000", 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0330000000000003e-06 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:31.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.077083+0000", 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001351323 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.077338+0000", 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025505299999999999 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.078292+0000", 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00095375099999999997 2024-09-04T21:54:31.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080448+0000", 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021567159999999999 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082012+0000", 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015639849999999999 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082182+0000", 2024-09-04T21:54:31.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169863 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082365+0000", 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018222600000000001 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24691 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:31.996 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.076136+0000", 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8064135750000005, 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0064383740000000002, 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24691 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:31.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076136+0000", 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076136+0000", 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076137+0000", 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.879999999999999e-07 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076143+0000", 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6949999999999997e-06 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076154+0000", 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1377e-05 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076161+0000", 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4819999999999997e-06 2024-09-04T21:54:31.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.079334+0000", 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031723060000000002 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.079434+0000", 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9853999999999996e-05 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.079840+0000", 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040624299999999998 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.081954+0000", 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021146339999999998 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082386+0000", 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043110199999999999 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082506+0000", 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120565 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082574+0000", 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8127999999999996e-05 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:32.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24692 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8063139630000009, 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0065395399999999999, 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24692 2024-09-04T21:54:32.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2699999999999999e-07 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076243+0000", 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8680000000000001e-06 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076257+0000", 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3923999999999999e-05 2024-09-04T21:54:32.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076262+0000", 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9110000000000001e-06 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080801+0000", 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00453908 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080893+0000", 2024-09-04T21:54:32.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2e-05 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.081333+0000", 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044013599999999999 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082612+0000", 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001279262 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082633+0000", 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0625e-05 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082711+0000", 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8319999999999996e-05 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082775+0000", 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3687000000000002e-05 2024-09-04T21:54:32.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:32.009 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:32.009 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:32.009 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:32.009 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:32.009 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:32.009 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:31.886+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:32.010 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:32.055 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.054+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:32.113 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:32.110+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:32.145 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:32.145 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:32.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:32.207 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:32.208 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:32.214 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.210+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.224 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:32.224 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:32.241 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:32.241 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:32.258 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:32.258 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:32.279 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:32.279 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:32.299 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:32.300 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:32.314 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:32.314 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:32.314 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.314+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.321 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:32.321 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:32.328 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:32.343 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:32.343 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-04T21:54:32.416 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.414+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.458 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:32.458 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:32.473 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:32.517 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.514+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.594 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:32.594 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:32.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:32.610 INFO:teuthology.orchestra.run.smithi112.stdout:2682 write 0x3b8c4f4 thru 0x3b986f0 (0xc1fd bytes) 2024-09-04T21:54:32.616 INFO:teuthology.orchestra.run.smithi112.stdout:2683 read 0x5d9af48 thru 0x5d9da53 (0x2b0c bytes) 2024-09-04T21:54:32.617 INFO:teuthology.orchestra.run.smithi112.stdout:2684 trunc from 0x7b2d4c5 to 0x8091906 2024-09-04T21:54:32.617 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:32.614+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:32.641 INFO:teuthology.orchestra.run.smithi112.stdout:2685 writesame 0xd85dd49 thru 0xd86086f data_size 0x1(0x2b27 bytes) 2024-09-04T21:54:32.641 INFO:teuthology.orchestra.run.smithi112.stdout:2686 write 0x4d8964b thru 0x4d8ab8a (0x1540 bytes) 2024-09-04T21:54:32.649 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.667 INFO:teuthology.orchestra.run.smithi112.stdout:2688 trunc from 0xd860870 to 0x2fcbba5 2024-09-04T21:54:32.718 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:32.714+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:32.747 INFO:teuthology.orchestra.run.smithi112.stdout:2689 punch from 0x12c2077 to 0x12c4e19, (0x2da2 bytes) 2024-09-04T21:54:32.747 INFO:teuthology.orchestra.run.smithi112.stdout:2692 read 0x1591111 thru 0x159df21 (0xce11 bytes) 2024-09-04T21:54:32.749 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.749 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.749 INFO:teuthology.orchestra.run.smithi112.stdout:2695 write 0x1c00fad thru 0x1c08064 (0x70b8 bytes) 2024-09-04T21:54:32.753 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:32.753 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:32.767 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:32.819 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.818+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.894 INFO:teuthology.orchestra.run.smithi112.stdout:2696 write 0x4f9d786 thru 0x4fad1d6 (0xfa51 bytes) 2024-09-04T21:54:32.899 INFO:teuthology.orchestra.run.smithi112.stdout:2698 punch from 0x19494d5 to 0x194e53d, (0x5068 bytes) 2024-09-04T21:54:32.900 INFO:teuthology.orchestra.run.smithi112.stdout:2700 punch from 0xe9b908 to 0xea1e90, (0x6588 bytes) 2024-09-04T21:54:32.900 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.900 INFO:teuthology.orchestra.run.smithi112.stdout:2705 writesame 0x1ce91b3 thru 0x1cf87a3 data_size 0x1f(0xf5f1 bytes) 2024-09-04T21:54:32.920 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:32.918+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:32.932 INFO:teuthology.orchestra.run.smithi112.stdout:2706 write 0xce47d82 thru 0xce50d1e (0x8f9d bytes) 2024-09-04T21:54:32.939 INFO:teuthology.orchestra.run.smithi112.stdout:2708 read 0x8f20f49 thru 0x8f24295 (0x334d bytes) 2024-09-04T21:54:32.939 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.939 INFO:teuthology.orchestra.run.smithi112.stdout:2710 write 0x85e633e thru 0x85f011c (0x9ddf bytes) 2024-09-04T21:54:32.941 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:32.941 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:32.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:32.961 INFO:teuthology.orchestra.run.smithi112.stdout:2712 write 0xdcdbedf thru 0xdce81ea (0xc30c bytes) 2024-09-04T21:54:32.968 INFO:teuthology.orchestra.run.smithi112.stdout:2713 write 0x3dcd82d thru 0x3ddc3cf (0xeba3 bytes) 2024-09-04T21:54:32.976 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.976 INFO:teuthology.orchestra.run.smithi112.stdout:2715 writesame 0xd4cc59f thru 0xd4d6a48 data_size 0x7(0xa4aa bytes) 2024-09-04T21:54:32.985 INFO:teuthology.orchestra.run.smithi112.stdout:2717 read 0x1a092e4 thru 0x1a14d35 (0xba52 bytes) 2024-09-04T21:54:32.986 INFO:teuthology.orchestra.run.smithi112.stdout:2718 read 0x562843a thru 0x562e287 (0x5e4e bytes) 2024-09-04T21:54:32.986 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.986 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:32.987 INFO:teuthology.orchestra.run.smithi112.stdout:2722 trunc from 0xdce81eb to 0xec44e4a 2024-09-04T21:54:32.997 INFO:teuthology.orchestra.run.smithi112.stdout:2725 punch from 0x2b063f4 to 0x2b08138, (0x1d44 bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2726 read 0xe62f20d thru 0xe63d556 (0xe34a bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2728 read 0x95ff932 thru 0x9607d14 (0x83e3 bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2729 read 0x90f6fa7 thru 0x90f7385 (0x3df bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2731 read 0xafb5d5c thru 0xafbf9d7 (0x9c7c bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2733 read 0x8866dba thru 0x886dd05 (0x6f4c bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2735 read 0x63878cf thru 0x6395292 (0xd9c4 bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2736 read 0x797ba6a thru 0x797fa49 (0x3fe0 bytes) 2024-09-04T21:54:32.998 INFO:teuthology.orchestra.run.smithi112.stdout:2737 write 0x4de6a1a thru 0x4deb3ba (0x49a1 bytes) 2024-09-04T21:54:33.000 INFO:teuthology.orchestra.run.smithi112.stdout:2738 writesame 0x4be4e77 thru 0x4be5b18 data_size 0x7(0xca2 bytes) 2024-09-04T21:54:33.006 INFO:teuthology.orchestra.run.smithi112.stdout:2739 punch from 0x725a42e to 0x7267f91, (0xdb63 bytes) 2024-09-04T21:54:33.021 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:33.018+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:33.023 INFO:teuthology.orchestra.run.smithi112.stdout:2740 trunc from 0xec44e4a to 0x488e0c2 2024-09-04T21:54:33.024 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:33.046 INFO:teuthology.orchestra.run.smithi112.stdout:2743 write 0xdc8002 thru 0xdc9148 (0x1147 bytes) 2024-09-04T21:54:33.095 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.095 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:33.109 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:33.110 INFO:teuthology.orchestra.run.smithi112.stdout:2744 read 0x2b5fc34 thru 0x2b614f9 (0x18c6 bytes) 2024-09-04T21:54:33.130 INFO:teuthology.orchestra.run.smithi112.stdout:2745 writesame 0x910592e thru 0x910d25e data_size 0x1(0x7931 bytes) 2024-09-04T21:54:33.130 INFO:teuthology.orchestra.run.smithi112.stdout:2747 punch from 0x3c3b5c1 to 0x3c45d99, (0xa7d8 bytes) 2024-09-04T21:54:33.130 INFO:teuthology.orchestra.run.smithi112.stdout:2749 read 0x7bd723e thru 0x7be294c (0xb70f bytes) 2024-09-04T21:54:33.130 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:33.126+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:33.146 INFO:teuthology.orchestra.run.smithi112.stdout:2750 write 0xea4fe42 thru 0xea559dd (0x5b9c bytes) 2024-09-04T21:54:33.160 INFO:teuthology.orchestra.run.smithi112.stdout:2752 trunc from 0xea559de to 0xa8547a4 2024-09-04T21:54:33.169 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.169 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:33.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-04T21:54:33.194 INFO:teuthology.orchestra.run.smithi112.stdout:2753 writesame 0xe181c8d thru 0xe18b485 data_size 0x1f(0x97f9 bytes) 2024-09-04T21:54:33.194 INFO:teuthology.orchestra.run.smithi112.stdout:2754 write 0x77d020c thru 0x77d0962 (0x757 bytes) 2024-09-04T21:54:33.199 INFO:teuthology.orchestra.run.smithi112.stdout:2755 write 0x4612032 thru 0x4618dfe (0x6dcd bytes) 2024-09-04T21:54:33.204 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:33.204 INFO:teuthology.orchestra.run.smithi112.stdout:2761 punch from 0xd6308f0 to 0xd631ad5, (0x11e5 bytes) 2024-09-04T21:54:33.205 INFO:teuthology.orchestra.run.smithi112.stdout:2762 write 0xb2065d9 thru 0xb20990d (0x3335 bytes) 2024-09-04T21:54:33.210 INFO:teuthology.orchestra.run.smithi112.stdout:2763 punch from 0x7f18d2 to 0x7fd746, (0xbe74 bytes) 2024-09-04T21:54:33.210 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:33.211 INFO:teuthology.orchestra.run.smithi112.stdout:2766 read 0x780d217 thru 0x781bbf1 (0xe9db bytes) 2024-09-04T21:54:33.211 INFO:teuthology.orchestra.run.smithi112.stdout:2767 read 0x708f12 thru 0x70facd (0x6bbc bytes) 2024-09-04T21:54:33.211 INFO:teuthology.orchestra.run.smithi112.stdout:2770 write 0xa7f52a5 thru 0xa8026e3 (0xd43f bytes) 2024-09-04T21:54:33.219 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:33.219 INFO:teuthology.orchestra.run.smithi112.stdout:2772 write 0x4b75bd0 thru 0x4b7ba48 (0x5e79 bytes) 2024-09-04T21:54:33.222 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:33.218+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:33.223 INFO:teuthology.orchestra.run.smithi112.stdout:2773 write 0x84829ed thru 0x8491a22 (0xf036 bytes) 2024-09-04T21:54:33.230 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:33.237 INFO:teuthology.orchestra.run.smithi112.stdout:2776 trunc from 0xe18b486 to 0x94c337a 2024-09-04T21:54:33.252 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:33.261 INFO:teuthology.orchestra.run.smithi112.stdout:2778 trunc from 0x94c337a to 0x2bc601e 2024-09-04T21:54:33.263 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.263 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:33.290 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:33.323 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:33.322+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:33.324 INFO:teuthology.orchestra.run.smithi112.stdout:2779 trunc from 0x2bc601e to 0x4c3ef52 2024-09-04T21:54:33.332 INFO:teuthology.orchestra.run.smithi112.stdout:2780 trunc from 0x4c3ef52 to 0x513f3e5 2024-09-04T21:54:33.339 INFO:teuthology.orchestra.run.smithi112.stdout:2781 read 0x177f8c3 thru 0x1784432 (0x4b70 bytes) 2024-09-04T21:54:33.341 INFO:teuthology.orchestra.run.smithi112.stdout:2782 read 0x2a8b509 thru 0x2a8d39e (0x1e96 bytes) 2024-09-04T21:54:33.342 INFO:teuthology.orchestra.run.smithi112.stdout:2783 trunc from 0x513f3e5 to 0x59717f9 2024-09-04T21:54:33.350 INFO:teuthology.orchestra.run.smithi112.stdout:2784 read 0x10be589 thru 0x10cab06 (0xc57e bytes) 2024-09-04T21:54:33.353 INFO:teuthology.orchestra.run.smithi112.stdout:2785 punch from 0x51543b3 to 0x515475d, (0x3aa bytes) 2024-09-04T21:54:33.353 INFO:teuthology.orchestra.run.smithi112.stdout:2786 punch from 0x20e3b0d to 0x20ef281, (0xb774 bytes) 2024-09-04T21:54:33.353 INFO:teuthology.orchestra.run.smithi112.stdout:2787 write 0x21e75c8 thru 0x21eae19 (0x3852 bytes) 2024-09-04T21:54:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:2788 write 0x616a825 thru 0x6177475 (0xcc51 bytes) 2024-09-04T21:54:33.385 INFO:teuthology.orchestra.run.smithi112.stdout:2789 write 0x8cfd92c thru 0x8d09441 (0xbb16 bytes) 2024-09-04T21:54:33.400 INFO:teuthology.orchestra.run.smithi112.stdout:2790 trunc from 0x8d09442 to 0x43f368d 2024-09-04T21:54:33.419 INFO:teuthology.orchestra.run.smithi112.stdout:2791 read 0xd5fe9c thru 0xd61e92 (0x1ff7 bytes) 2024-09-04T21:54:33.424 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:33.422+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:33.438 INFO:teuthology.orchestra.run.smithi112.stdout:2792 write 0xa613bfc thru 0xa617027 (0x342c bytes) 2024-09-04T21:54:33.447 INFO:teuthology.orchestra.run.smithi112.stdout:2793 writesame 0x85c3e64 thru 0x85d220a data_size 0x1(0xe3a7 bytes) 2024-09-04T21:54:33.450 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.450 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:33.465 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:33.495 INFO:teuthology.orchestra.run.smithi112.stdout:2794 read 0x2e9c946 thru 0x2eab4f7 (0xebb2 bytes) 2024-09-04T21:54:33.495 INFO:teuthology.orchestra.run.smithi112.stdout:2796 writesame 0x452117 thru 0x45ab12 data_size 0x2(0x89fc bytes) 2024-09-04T21:54:33.525 INFO:teuthology.orchestra.run.smithi112.stdout:2797 writesame 0xe12a101 thru 0xe13781c data_size 0x4e(0xd71c bytes) 2024-09-04T21:54:33.526 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:33.526 INFO:teuthology.orchestra.run.smithi112.stdout:2799 writesame 0xc038584 thru 0xc03f782 data_size 0xb(0x71ff bytes) 2024-09-04T21:54:33.526 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:33.522+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:33.539 INFO:teuthology.orchestra.run.smithi112.stdout:2800 trunc from 0xe13781d to 0xaa85e19 2024-09-04T21:54:33.557 INFO:teuthology.orchestra.run.smithi112.stdout:2801 punch from 0x75f9ae6 to 0x75fd051, (0x356b bytes) 2024-09-04T21:54:33.558 INFO:teuthology.orchestra.run.smithi112.stdout:2802 read 0x646834d thru 0x6468ea9 (0xb5d bytes) 2024-09-04T21:54:33.558 INFO:teuthology.orchestra.run.smithi112.stdout:2803 write 0x868d68e thru 0x869725b (0x9bce bytes) 2024-09-04T21:54:33.560 INFO:teuthology.orchestra.run.smithi112.stdout:2804 read 0x5e1c0cc thru 0x5e2af8f (0xeec4 bytes) 2024-09-04T21:54:33.575 INFO:teuthology.orchestra.run.smithi112.stdout:2805 writesame 0xed927f3 thru 0xed9f967 data_size 0x1d(0xd175 bytes) 2024-09-04T21:54:33.576 INFO:teuthology.orchestra.run.smithi112.stdout:2806 writesame 0xcacbf22 thru 0xcad37dd data_size 0x2(0x78bc bytes) 2024-09-04T21:54:33.591 INFO:teuthology.orchestra.run.smithi112.stdout:2807 writesame 0x999a2c7 thru 0x999f828 data_size 0x3(0x5562 bytes) 2024-09-04T21:54:33.595 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.0 to 1 (10000) 2024-09-04T21:54:33.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:54:33.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-04T21:54:33.627 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:33.626+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:33.635 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.636 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:33.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:33.744 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:33.742+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:33.822 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.823 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:33.828 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:33.826+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:33.837 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:33.929 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:33.926+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:33.997 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:33.997 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:34.010 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:34.053 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:34.050+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:34.131 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:34.126+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:34.143 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:34.144 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:34.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:34.231 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:34.226+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:34.283 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:34.283 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:34.298 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:34.332 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:34.330+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:34.363 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:34.428 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:34.428 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:34.433 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:34.429+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:34.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:34.520 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:34.534 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:34.530+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:34.537 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:34.537 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:34.553 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:34.554 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:34.573 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:34.573 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:34.594 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:54:34.595 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:34.595 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.2 to 1 (10000) 2024-09-04T21:54:34.595 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:34.617 INFO:teuthology.orchestra.run.smithi112.stdout:2810 trunc from 0xed9f968 to 0xe7ef93c 2024-09-04T21:54:34.623 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:34.623 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:34.623 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:34.623 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:34.624 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:34.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-04T21:54:34.627 INFO:teuthology.orchestra.run.smithi112.stdout:2811 write 0xb410013 thru 0xb41cb09 (0xcaf7 bytes) 2024-09-04T21:54:34.635 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:34.633+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:34.636 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:34.636 INFO:teuthology.orchestra.run.smithi112.stdout:2814 punch from 0xa45f343 to 0xa467a42, (0x86ff bytes) 2024-09-04T21:54:34.636 INFO:teuthology.orchestra.run.smithi112.stdout:2816 punch from 0x2f75dbe to 0x2f7fea4, (0xa0e6 bytes) 2024-09-04T21:54:34.636 INFO:teuthology.orchestra.run.smithi112.stdout:2817 read 0x2273f01 thru 0x2275472 (0x1572 bytes) 2024-09-04T21:54:34.637 INFO:teuthology.orchestra.run.smithi112.stdout:2818 read 0xe022520 thru 0xe0292b5 (0x6d96 bytes) 2024-09-04T21:54:34.639 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:34.645 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:34.645 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:34.647 INFO:teuthology.orchestra.run.smithi112.stdout:2819 trunc from 0xe7ef93c to 0x722da3c 2024-09-04T21:54:34.663 INFO:teuthology.orchestra.run.smithi112.stdout:2820 punch from 0x4bc199 to 0x4befc5, (0x2e2c bytes) 2024-09-04T21:54:34.663 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:34.663 INFO:teuthology.orchestra.run.smithi112.stdout:2822 read 0x4aef89b thru 0x4af6749 (0x6eaf bytes) 2024-09-04T21:54:34.663 INFO:teuthology.orchestra.run.smithi112.stdout:2823 write 0x3fbb5fe thru 0x3fc26f3 (0x70f6 bytes) 2024-09-04T21:54:34.668 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:34.668 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:34.670 INFO:teuthology.orchestra.run.smithi112.stdout:2824 read 0x8d1b2 thru 0x8d318 (0x167 bytes) 2024-09-04T21:54:34.670 INFO:teuthology.orchestra.run.smithi112.stdout:2826 punch from 0x19179b3 to 0x191b05e, (0x36ab bytes) 2024-09-04T21:54:34.670 INFO:teuthology.orchestra.run.smithi112.stdout:2827 read 0x6e48e9e thru 0x6e49039 (0x19c bytes) 2024-09-04T21:54:34.678 INFO:teuthology.orchestra.run.smithi112.stdout:2828 write 0x8849694 thru 0x884df3a (0x48a7 bytes) 2024-09-04T21:54:34.684 INFO:teuthology.orchestra.run.smithi112.stdout:2830 read 0x632def9 thru 0x6332e4e (0x4f56 bytes) 2024-09-04T21:54:34.696 INFO:teuthology.orchestra.run.smithi112.stdout:2831 writesame 0xc27de93 thru 0xc28cb88 data_size 0x31(0xecf6 bytes) 2024-09-04T21:54:34.700 INFO:teuthology.orchestra.run.smithi112.stdout:2832 trunc from 0xc28cb89 to 0xe8abe6c 2024-09-04T21:54:34.713 INFO:teuthology.orchestra.run.smithi112.stdout:2833 writesame 0xec67b6d thru 0xec77b13 data_size 0x1(0xffa7 bytes) 2024-09-04T21:54:34.713 INFO:teuthology.orchestra.run.smithi112.stdout:2834 read 0xd9fc647 thru 0xda09502 (0xcebc bytes) 2024-09-04T21:54:34.713 INFO:teuthology.orchestra.run.smithi112.stdout:2835 read 0x296dbb5 thru 0x2970079 (0x24c5 bytes) 2024-09-04T21:54:34.713 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:34.713 INFO:teuthology.orchestra.run.smithi112.stdout:2837 writesame 0xb1138c7 thru 0xb11d6f9 data_size 0x1(0x9e33 bytes) 2024-09-04T21:54:34.728 INFO:teuthology.orchestra.run.smithi112.stdout:2838 writesame 0x327c0cc thru 0x327d9e3 data_size 0x2c(0x1918 bytes) 2024-09-04T21:54:34.732 INFO:teuthology.orchestra.run.smithi112.stdout:2842 read 0x66c12f3 thru 0x66d0dec (0xfafa bytes) 2024-09-04T21:54:34.732 INFO:teuthology.orchestra.run.smithi112.stdout:2843 punch from 0x4fa7d34 to 0x4fad741, (0x5a0d bytes) 2024-09-04T21:54:34.732 INFO:teuthology.orchestra.run.smithi112.stdout:2845 write 0x4138a8c thru 0x41466a4 (0xdc19 bytes) 2024-09-04T21:54:34.735 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:34.730+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:34.738 INFO:teuthology.orchestra.run.smithi112.stdout:2846 punch from 0xa239db2 to 0xa2401e4, (0x6432 bytes) 2024-09-04T21:54:34.738 INFO:teuthology.orchestra.run.smithi112.stdout:2847 writesame 0x7dd4c50 thru 0x7de0ce4 data_size 0x1(0xc095 bytes) 2024-09-04T21:54:34.756 INFO:teuthology.orchestra.run.smithi112.stdout:2849 punch from 0xd0f77c6 to 0xd101465, (0x9c9f bytes) 2024-09-04T21:54:34.756 INFO:teuthology.orchestra.run.smithi112.stdout:2851 punch from 0x4d72726 to 0x4d7d98b, (0xb265 bytes) 2024-09-04T21:54:34.766 INFO:teuthology.orchestra.run.smithi112.stdout:2852 trunc from 0xec77b14 to 0x82f6581 2024-09-04T21:54:34.793 INFO:teuthology.orchestra.run.smithi112.stdout:2853 write 0xcc0517b thru 0xcc07c01 (0x2a87 bytes) 2024-09-04T21:54:34.801 INFO:teuthology.orchestra.run.smithi112.stdout:2855 read 0x82c5c60 thru 0x82d083c (0xabdd bytes) 2024-09-04T21:54:34.801 INFO:teuthology.orchestra.run.smithi112.stdout:2856 write 0x2bf61ba thru 0x2bfb83b (0x5682 bytes) 2024-09-04T21:54:34.803 INFO:teuthology.orchestra.run.smithi112.stdout:2857 write 0x855c17c thru 0x855d0a8 (0xf2d bytes) 2024-09-04T21:54:34.807 INFO:teuthology.orchestra.run.smithi112.stdout:2859 read 0x8d16dad thru 0x8d2089d (0x9af1 bytes) 2024-09-04T21:54:34.807 INFO:teuthology.orchestra.run.smithi112.stdout:2860 write 0x4ce0ee thru 0x4ddefc (0xfe0f bytes) 2024-09-04T21:54:34.809 INFO:teuthology.orchestra.run.smithi112.stdout:2861 writesame 0x5f0065e thru 0x5f08a71 data_size 0x4(0x8414 bytes) 2024-09-04T21:54:34.816 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:34.816 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:34.816 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:34.816 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:34.816 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:34.817 INFO:teuthology.orchestra.run.smithi112.stdout:2862 punch from 0xc7f6f17 to 0xc7fa1e4, (0x32cd bytes) 2024-09-04T21:54:34.817 INFO:teuthology.orchestra.run.smithi112.stdout:2864 read 0x502310 thru 0x510e1d (0xeb0e bytes) 2024-09-04T21:54:34.818 INFO:teuthology.orchestra.run.smithi112.stdout:2865 read 0x576478b thru 0x576a167 (0x59dd bytes) 2024-09-04T21:54:34.821 INFO:teuthology.orchestra.run.smithi112.stdout:2866 trunc from 0xcc07c02 to 0xab7396a 2024-09-04T21:54:34.832 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:34.835 INFO:teuthology.orchestra.run.smithi112.stdout:2867 writesame 0x4460cdb thru 0x44650f6 data_size 0x6(0x441c bytes) 2024-09-04T21:54:34.839 INFO:teuthology.orchestra.run.smithi112.stdout:2868 read 0x58c6493 thru 0x58ca0f5 (0x3c63 bytes) 2024-09-04T21:54:34.847 INFO:teuthology.orchestra.run.smithi112.stdout:2870 write 0xba77af0 thru 0xba86d8b (0xf29c bytes) 2024-09-04T21:54:34.852 INFO:teuthology.orchestra.run.smithi112.stdout:2871 read 0x7c65cde thru 0x7c672ce (0x15f1 bytes) 2024-09-04T21:54:34.852 INFO:teuthology.orchestra.run.smithi112.stdout:2872 punch from 0xa361ef7 to 0xa36779a, (0x58a3 bytes) 2024-09-04T21:54:34.853 INFO:teuthology.orchestra.run.smithi112.stdout:2876 read 0x1937531 thru 0x1946c36 (0xf706 bytes) 2024-09-04T21:54:34.853 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:34.853 INFO:teuthology.orchestra.run.smithi112.stdout:2878 write 0x2e7412b thru 0x2e793a3 (0x5279 bytes) 2024-09-04T21:54:34.857 INFO:teuthology.orchestra.run.smithi112.stdout:2879 writesame 0x16cdb30 thru 0x16db80c data_size 0x2f(0xdcdd bytes) 2024-09-04T21:54:34.882 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:34.881+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:34.934 INFO:teuthology.orchestra.run.smithi112.stdout:2881 trunc from 0xba86d8c to 0x21e5f7a 2024-09-04T21:54:34.936 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:34.930+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:34.963 INFO:teuthology.orchestra.run.smithi112.stdout:2882 writesame 0x451449e thru 0x4514c1f data_size 0x2(0x782 bytes) 2024-09-04T21:54:34.972 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19512 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 983040~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636624+0000", 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334975241, 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025024366999999999, 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19512 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636624+0000", 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636624+0000", 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636625+0000", 2024-09-04T21:54:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636652+0000", 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7061000000000001e-05 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636663+0000", 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1080000000000001e-05 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636671+0000", 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.503e-06 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661350+0000", 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024679570000000001 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661470+0000", 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119564 2024-09-04T21:54:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661649+0000", 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000178889 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19513 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1146880~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636704+0000", 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334895915000001, 2024-09-04T21:54:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025639926, 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19513 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636704+0000", 2024-09-04T21:54:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636704+0000", 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636705+0000", 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636709+0000", 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8759999999999998e-06 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636724+0000", 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5432999999999999e-05 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636731+0000", 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.661777+0000", 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025045290000000001 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662043+0000", 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000265868 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662344+0000", 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030111599999999998 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19514 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1310720~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636772+0000", 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334828131, 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025968494000000002, 2024-09-04T21:54:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19514 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636772+0000", 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636772+0000", 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636773+0000", 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1030000000000001e-06 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636776+0000", 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7529999999999999e-06 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636800+0000", 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3728999999999999e-05 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636808+0000", 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9859999999999994e-06 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662449+0000", 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025640616000000001 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662580+0000", 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013143399999999999 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662740+0000", 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015987299999999999 2024-09-04T21:54:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19515 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1556480~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636821+0000", 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334778727, 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026552834000000001, 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19515 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636821+0000", 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636821+0000", 2024-09-04T21:54:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636822+0000", 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7000000000000003e-07 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636825+0000", 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4960000000000001e-06 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636835+0000", 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2930000000000006e-06 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636842+0000", 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0169999999999999e-06 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.662841+0000", 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025999586000000002 2024-09-04T21:54:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663075+0000", 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000233289 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663374+0000", 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029918300000000001 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19516 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1720320~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636852+0000", 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334747678999999, 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027108402, 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19516 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636852+0000", 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636852+0000", 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636853+0000", 2024-09-04T21:54:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3699999999999999e-07 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636856+0000", 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1099999999999999e-06 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636865+0000", 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636872+0000", 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8859999999999998e-06 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663502+0000", 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026630233999999999 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663654+0000", 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151886 2024-09-04T21:54:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.663960+0000", 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030664899999999998 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19517 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 1884160~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636881+0000", 2024-09-04T21:54:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334718383, 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027694871999999999, 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19517 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636881+0000", 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636881+0000", 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636882+0000", 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636885+0000", 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9239999999999999e-06 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636893+0000", 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6719999999999993e-06 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636901+0000", 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1500000000000002e-06 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664058+0000", 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027157642999999999 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664285+0000", 2024-09-04T21:54:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022626600000000001 2024-09-04T21:54:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664576+0000", 2024-09-04T21:54:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029158400000000001 2024-09-04T21:54:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19518 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2293760~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636947+0000", 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334653003, 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028244814, 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19518 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636947+0000", 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636947+0000", 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636948+0000", 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0839999999999999e-06 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636952+0000", 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9829999999999998e-06 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636963+0000", 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1715999999999999e-05 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636971+0000", 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664671+0000", 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027699745000000001 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.664913+0000", 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024215599999999999 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665191+0000", 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027864699999999998 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19519 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2457600~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334617985, 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028816913999999999, 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19519 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636982+0000", 2024-09-04T21:54:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636986+0000", 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.636994+0000", 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8599999999999999e-06 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637002+0000", 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1489999999999997e-06 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665282+0000", 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028280784999999999 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665515+0000", 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232449 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665799+0000", 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028375699999999999 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19520 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2621440~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637079+0000", 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334520791999999, 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029314778, 2024-09-04T21:54:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19520 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637079+0000", 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637079+0000", 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637080+0000", 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.097e-06 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637083+0000", 2024-09-04T21:54:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4429999999999999e-06 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637093+0000", 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8789999999999998e-06 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637100+0000", 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1330000000000001e-06 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.665890+0000", 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.028789848999999999 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666123+0000", 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232642 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666394+0000", 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027073499999999998 2024-09-04T21:54:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19521 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 2867200~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637111+0000", 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334488684, 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029880898999999999, 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19521 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637111+0000", 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637111+0000", 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637112+0000", 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9699999999999995e-07 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637115+0000", 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3629999999999998e-06 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637124+0000", 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8769999999999999e-06 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637131+0000", 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0230000000000004e-06 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666478+0000", 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029346819 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666717+0000", 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023884899999999999 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.666992+0000", 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000275271 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19522 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3031040~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637175+0000", 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334424713000001, 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030447393, 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19522 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637175+0000", 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637175+0000", 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637176+0000", 2024-09-04T21:54:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9300000000000006e-07 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637180+0000", 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7629999999999998e-06 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637190+0000", 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9469999999999998e-06 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637199+0000", 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9490000000000002e-06 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667082+0000", 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.029883581999999999 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667331+0000", 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024896800000000002 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667622+0000", 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000291191 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19523 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3194880~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637210+0000", 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334389898, 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030852570999999999, 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19523 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637210+0000", 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637210+0000", 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637211+0000", 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-06 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637214+0000", 2024-09-04T21:54:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.366e-06 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637223+0000", 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8429999999999999e-06 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637230+0000", 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0330000000000003e-06 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667728+0000", 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030497570000000002 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.667887+0000", 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015933700000000001 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668062+0000", 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017534500000000001 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19524 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3604480~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637252+0000", 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334347647, 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.031460546999999998, 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19524 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637252+0000", 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637252+0000", 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637253+0000", 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637278+0000", 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5632000000000001e-05 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637290+0000", 2024-09-04T21:54:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2055999999999999e-05 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637298+0000", 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9799999999999998e-06 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668168+0000", 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.030870056 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668431+0000", 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026247800000000002 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668713+0000", 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028162100000000002 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19525 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3768320~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637346+0000", 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334253401, 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.031987768999999999, 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19525 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637346+0000", 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637346+0000", 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637347+0000", 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:54:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637351+0000", 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1699999999999999e-06 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637362+0000", 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0363000000000001e-05 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637369+0000", 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2799999999999998e-06 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.668802+0000", 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.031433097 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669051+0000", 2024-09-04T21:54:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024841199999999999 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669334+0000", 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028349700000000001 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19526 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 3932160~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637380+0000", 2024-09-04T21:54:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334219399, 2024-09-04T21:54:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032563683000000003, 2024-09-04T21:54:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19526 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637380+0000", 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637380+0000", 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637381+0000", 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0759999999999999e-06 2024-09-04T21:54:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637384+0000", 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.174e-06 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637393+0000", 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637400+0000", 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0260000000000002e-06 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669425+0000", 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032024997999999999 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669670+0000", 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024425500000000003 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.669944+0000", 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000274338 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19527 3.1c 3:3e1b5884:::rbd_data.1188cf8351f7.000000000000000b:b [read 4177920~16384] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637410+0000", 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334189537, 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032943358999999998, 2024-09-04T21:54:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19527 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637410+0000", 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637410+0000", 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637411+0000", 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637414+0000", 2024-09-04T21:54:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0460000000000001e-06 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637422+0000", 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6160000000000002e-06 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637429+0000", 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0870000000000001e-06 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670050+0000", 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032620709999999997 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670186+0000", 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135921 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670353+0000", 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016733899999999999 2024-09-04T21:54:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19541 3.c 3:35e97e9b:::rbd_data.118892c64562.0000000000000033:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 950272] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.637439+0000", 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.334160432000001, 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.038452027, 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19541 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637439+0000", 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637439+0000", 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637440+0000", 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4700000000000001e-07 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637684+0000", 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024439200000000001 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637701+0000", 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6356000000000001e-05 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.637711+0000", 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0219e-05 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670035+0000", 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.032324031000000003 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670393+0000", 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035815700000000001 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.671203+0000", 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00080962700000000005 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675556+0000", 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043532600000000003 2024-09-04T21:54:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675594+0000", 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7509000000000003e-05 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675776+0000", 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000182981 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.675891+0000", 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011474800000000001 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:19551 3.1c 3:3bdbea5f:::rbd_data.118892c64562.0000000000000038:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 950272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:20.646954+0000", 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.324645967, 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.027736934000000001, 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 19551 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646954+0000", 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646954+0000", 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646956+0000", 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0140000000000001e-06 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646966+0000", 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9420000000000007e-06 2024-09-04T21:54:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.646991+0000", 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4986000000000001e-05 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.647005+0000", 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4036000000000001e-05 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670401+0000", 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023396107999999999 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670580+0000", 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017891200000000001 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.670852+0000", 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027222200000000002 2024-09-04T21:54:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674314+0000", 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034617480000000002 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674348+0000", 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4668000000000001e-05 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674568+0000", 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021935699999999999 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:20.674691+0000", 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000122941 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.5298342849999997, 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:54:35.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:54:35.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:54:35.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.3452932299999993, 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.069 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:54:35.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.074 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.075 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.075 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.075 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.075 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:35.077 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:35.077 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:35.073+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2883 write 0x887ee76 thru 0x888d86a (0xe9f5 bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2885 write 0x82801f9 thru 0x8283696 (0x349e bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2886 writesame 0xed33ef7 thru 0xed34229 data_size 0x7(0x333 bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2889 writesame 0x9563e58 thru 0x957207b data_size 0x29(0xe224 bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2893 read 0x6640c1f thru 0x664f3c8 (0xe7aa bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2895 write 0xb33d328 thru 0xb341ecb (0x4ba4 bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2896 read 0xc7dbfb0 thru 0xc7e7814 (0xb865 bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2897 write 0x182953c thru 0x182dfe7 (0x4aac bytes) 2024-09-04T21:54:35.078 INFO:teuthology.orchestra.run.smithi112.stdout:2900 read 0x95beaf6 thru 0x95cd644 (0xeb4f bytes) 2024-09-04T21:54:35.079 INFO:teuthology.orchestra.run.smithi112.stdout:2902 write 0xf12136 thru 0xf13326 (0x11f1 bytes) 2024-09-04T21:54:35.098 INFO:teuthology.orchestra.run.smithi112.stdout:2903 trunc from 0xed3422a to 0x1680ce3 2024-09-04T21:54:35.127 INFO:teuthology.orchestra.run.smithi112.stdout:2904 read 0x4e3b92 thru 0x4e8853 (0x4cc2 bytes) 2024-09-04T21:54:35.144 INFO:teuthology.orchestra.run.smithi112.stdout:2905 trunc from 0x1680ce3 to 0x83dd86c 2024-09-04T21:54:35.156 INFO:teuthology.orchestra.run.smithi112.stdout:2906 writesame 0x37c906 thru 0x380fde data_size 0x1(0x46d9 bytes) 2024-09-04T21:54:35.177 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:35.174+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:35.262 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:35.262 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:35.275 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:35.275 INFO:teuthology.orchestra.run.smithi112.stdout:2911 punch from 0x493e8fa to 0x4942280, (0x3986 bytes) 2024-09-04T21:54:35.275 INFO:teuthology.orchestra.run.smithi112.stdout:2912 read 0x17986fb thru 0x17a65da (0xdee0 bytes) 2024-09-04T21:54:35.275 INFO:teuthology.orchestra.run.smithi112.stdout:2914 read 0x27ca13a thru 0x27ced0a (0x4bd1 bytes) 2024-09-04T21:54:35.277 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:35.279 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:35.274+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:35.303 INFO:teuthology.orchestra.run.smithi112.stdout:2915 writesame 0xdedba6a thru 0xdedbdd8 data_size 0x1(0x36f bytes) 2024-09-04T21:54:35.303 INFO:teuthology.orchestra.run.smithi112.stdout:2916 read 0x52f3d70 thru 0x5301e01 (0xe092 bytes) 2024-09-04T21:54:35.303 INFO:teuthology.orchestra.run.smithi112.stdout:2917 clone 14 order 21 su 262144 sc 6 2024-09-04T21:54:35.380 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:35.374+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:35.401 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:35.401 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:35.401 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:35.401 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:35.401 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:35.414 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:35.481 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:35.477+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:35.543 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:35.543 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:35.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:35.543 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1045 3.10 3:08ed8dc3:::rbd_data.11882e693cd0.000000000000002f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 33.625786546999997, 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054299810000000004, 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.544 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1045 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912474+0000", 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912477+0000", 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8369999999999999e-06 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912492+0000", 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.428e-05 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912529+0000", 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7673000000000003e-05 2024-09-04T21:54:35.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912547+0000", 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7689e-05 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912710+0000", 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162526 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.912993+0000", 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028371199999999998 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.913324+0000", 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033046099999999999 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.915482+0000", 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021584989999999999 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917665+0000", 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021827449999999998 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917778+0000", 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000113063 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:01.917904+0000", 2024-09-04T21:54:35.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012549600000000001 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:1101 3.a 3:51afe48d:::rbd_header.1188e7c968d9:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 33.117930504999997, 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0067943780000000002, 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1101 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420330+0000", 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420333+0000", 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6169999999999998e-06 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420347+0000", 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-05 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420381+0000", 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4122999999999998e-05 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420398+0000", 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6286999999999998e-05 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420568+0000", 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017088199999999999 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.420589+0000", 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0513000000000002e-05 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424142+0000", 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035527359999999999 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424352+0000", 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021066300000000001 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.424715+0000", 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000362233 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426277+0000", 2024-09-04T21:54:35.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015626069999999999 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.426969+0000", 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069220800000000004 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427060+0000", 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1057000000000006e-05 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:02.427124+0000", 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3522000000000006e-05 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7519 3.10 3:0bafe686:::rbd_data.1188b1294ada.0000000000000046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.043231146, 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054280980000000001, 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.557 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7519 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495029+0000", 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495033+0000", 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7299999999999999e-06 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495047+0000", 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4066000000000001e-05 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495089+0000", 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2700000000000001e-05 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495112+0000", 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2965000000000001e-05 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495230+0000", 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117581 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.495238+0000", 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.388e-06 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498788+0000", 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035494939999999998 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498872+0000", 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4190999999999994e-05 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.498983+0000", 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110739 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.499844+0000", 2024-09-04T21:54:35.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086077500000000004 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500358+0000", 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051428599999999997 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500414+0000", 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5689999999999997e-05 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.500457+0000", 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3492999999999998e-05 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7675 3.a 3:52665b53:::rbd_data.1188b1294ada.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1179648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 27.802439732, 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057294420000000004, 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.564 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7675 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735820+0000", 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.735829+0000", 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8079999999999994e-06 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736259+0000", 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042982200000000001 2024-09-04T21:54:35.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736276+0000", 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7124000000000001e-05 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736287+0000", 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0899e-05 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736380+0000", 2024-09-04T21:54:35.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3304e-05 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736530+0000", 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014963300000000001 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.736871+0000", 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034072099999999999 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.739225+0000", 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023541030000000002 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741419+0000", 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021947189999999999 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741495+0000", 2024-09-04T21:54:35.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5087999999999997e-05 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:07.741550+0000", 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5220999999999999e-05 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:7875 3.a 3:562be18b:::rbd_data.1188b1294ada.0000000000000001:head [stat,truncate 262144] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:35.570 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.959118844999999, 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052294439999999998, 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 7875 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:35.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579141+0000", 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579142+0000", 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0500000000000002e-07 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579147+0000", 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1120000000000004e-06 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579154+0000", 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.097e-06 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.579159+0000", 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4870000000000004e-06 2024-09-04T21:54:35.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581460+0000", 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023010230000000001 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581541+0000", 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0939999999999994e-05 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.581739+0000", 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019828000000000001 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584176+0000", 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002437051 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584259+0000", 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2831999999999996e-05 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584320+0000", 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1204000000000006e-05 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:08.584371+0000", 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0612999999999998e-05 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14753 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.088543+0000", 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.449716896999998, 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056008339999999998, 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14753 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.577 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088543+0000", 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088543+0000", 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088545+0000", 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.73e-06 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088553+0000", 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4230000000000005e-06 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088574+0000", 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0749000000000001e-05 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.088584+0000", 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.027e-05 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.090476+0000", 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018915679999999999 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.090753+0000", 2024-09-04T21:54:35.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000276913 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.091809+0000", 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001056162 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.093605+0000", 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017957019999999999 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.093944+0000", 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033892699999999999 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.094033+0000", 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9538999999999998e-05 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.094144+0000", 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110851 2024-09-04T21:54:35.582 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14768 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.160622+0000", 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.377638335, 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0069038479999999998, 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14768 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160622+0000", 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160622+0000", 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160625+0000", 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4070000000000002e-06 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160641+0000", 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5536000000000002e-05 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160671+0000", 2024-09-04T21:54:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0660999999999997e-05 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160693+0000", 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1716e-05 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160837+0000", 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014385999999999999 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161062+0000", 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022511499999999999 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.162272+0000", 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001209578 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167334+0000", 2024-09-04T21:54:35.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050621470000000003 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167356+0000", 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2351999999999999e-05 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167455+0000", 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8392000000000003e-05 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167526+0000", 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1083999999999999e-05 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14769 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.377484898999999, 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.008898853, 2024-09-04T21:54:35.589 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14769 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160777+0000", 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.486e-06 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160786+0000", 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3740000000000004e-06 2024-09-04T21:54:35.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160806+0000", 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0165000000000001e-05 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160822+0000", 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6203000000000001e-05 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163870+0000", 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003047812 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.164137+0000", 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000266643 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.164872+0000", 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073519099999999995 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167623+0000", 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027509129999999998 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169518+0000", 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001895274 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169593+0000", 2024-09-04T21:54:35.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4270000000000006e-05 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169674+0000", 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1521999999999996e-05 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14770 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:35.595 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.377220158, 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0090832199999999995, 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14770 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:35.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161042+0000", 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4930000000000001e-06 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161054+0000", 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1466e-05 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161079+0000", 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5114999999999999e-05 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161097+0000", 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8315999999999999e-05 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.165726+0000", 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0046287849999999998 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.165869+0000", 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014293300000000001 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.166506+0000", 2024-09-04T21:54:35.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063653199999999998 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169697+0000", 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031909410000000001 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169867+0000", 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170478 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170053+0000", 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018593099999999999 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170123+0000", 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0229999999999997e-05 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14771 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.374927450000001, 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089354540000000007, 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14771 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.602 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163335+0000", 2024-09-04T21:54:35.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7030000000000002e-06 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163350+0000", 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4523e-05 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163387+0000", 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6817999999999999e-05 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163408+0000", 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1715000000000001e-05 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167677+0000", 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042681560000000004 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167880+0000", 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020378 2024-09-04T21:54:35.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.168602+0000", 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00072149099999999999 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170348+0000", 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017465849999999999 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172076+0000", 2024-09-04T21:54:35.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001727836 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172205+0000", 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012882700000000001 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172268+0000", 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3020000000000003e-05 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.607 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:23212 3.0 e41/38 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head v 41'114, mlcod=39'112)", 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.219901981, 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0092641379999999999, 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318361+0000", 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:35.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322409+0000", 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040474400000000002 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322451+0000", 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2240999999999998e-05 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322473+0000", 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1676e-05 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322671+0000", 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019802099999999999 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322713+0000", 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2645000000000003e-05 2024-09-04T21:54:35.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.327478+0000", 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047643989999999999 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.327622+0000", 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014443 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:23214 3.1e e41/22 3:7a40d9fa:::rbd_data.118892c64562.0000000000000009:head v 41'75, mlcod=41'73)", 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.205078686, 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088767499999999992, 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:35.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333184+0000", 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9469999999999999e-06 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335653+0000", 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024681690000000001 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335684+0000", 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1112000000000003e-05 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335697+0000", 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3329999999999999e-05 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335861+0000", 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016441900000000001 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335929+0000", 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7953999999999993e-05 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.341970+0000", 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006040803 2024-09-04T21:54:35.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.342058+0000", 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8016000000000006e-05 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:23926 3.a 3:547e81fb:::rbd_data.118892c64562.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1622016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.790103630000001, 2024-09-04T21:54:35.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010368176, 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 23926 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:35.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748159+0000", 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8729999999999999e-06 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750340+0000", 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021802879999999998 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750372+0000", 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2758e-05 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750395+0000", 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2368999999999999e-05 2024-09-04T21:54:35.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750558+0000", 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016326000000000001 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750850+0000", 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000292117 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.751390+0000", 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054000999999999995 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.757506+0000", 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061158460000000003 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758289+0000", 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078254599999999996 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758407+0000", 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011886400000000001 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758525+0000", 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011724500000000001 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24315 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.924893321000001, 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074709240000000003, 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24315 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613370+0000", 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.49e-06 2024-09-04T21:54:35.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613385+0000", 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5112e-05 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613421+0000", 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5246000000000001e-05 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613438+0000", 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7051999999999999e-05 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613585+0000", 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014730799999999999 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613612+0000", 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7212e-05 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.617673+0000", 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040609959999999999 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.617898+0000", 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022498400000000001 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.618232+0000", 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033338999999999998 2024-09-04T21:54:35.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.619944+0000", 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001712746 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620686+0000", 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074206200000000004 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620774+0000", 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7249000000000004e-05 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620838+0000", 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4077000000000003e-05 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24372 3.a 3:547e81fb:::rbd_data.118892c64562.0000000000000002:head [stat,truncate 1081344] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:35.630 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.855865974, 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088444000000000005, 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24372 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682398+0000", 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.924e-06 2024-09-04T21:54:35.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682412+0000", 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3782e-05 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682458+0000", 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5738000000000002e-05 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682483+0000", 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5313e-05 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682668+0000", 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018540200000000001 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682935+0000", 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000266412 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.683466+0000", 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053094700000000004 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.689562+0000", 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060959769999999998 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.690972+0000", 2024-09-04T21:54:35.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014100499999999999 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691102+0000", 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013006399999999999 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691239+0000", 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013679100000000001 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:24378 3.0 e42/38 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head v 42'122, mlcod=41'119)", 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.685916+0000", 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.852344304000001, 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052256239999999999, 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685916+0000", 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685916+0000", 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685918+0000", 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700000000000001e-06 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685935+0000", 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7169000000000001e-05 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685966+0000", 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0889000000000002e-05 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.685985+0000", 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8258999999999999e-05 2024-09-04T21:54:35.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.686215+0000", 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023046699999999999 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.686268+0000", 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3424999999999998e-05 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691024+0000", 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047552200000000001 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691141+0000", 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117725 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24689 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:35.641 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.075516+0000", 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.462744186, 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052406229999999998, 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24689 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075516+0000", 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075516+0000", 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075517+0000", 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3430000000000001e-06 2024-09-04T21:54:35.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075522+0000", 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0950000000000003e-06 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075535+0000", 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2975e-05 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075541+0000", 2024-09-04T21:54:35.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4700000000000001e-06 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075608+0000", 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7175000000000007e-05 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075700+0000", 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2453000000000004e-05 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076089+0000", 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000388496 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080300+0000", 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042111060000000001 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080487+0000", 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018713700000000001 2024-09-04T21:54:35.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080642+0000", 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015494400000000001 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080757+0000", 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114429 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24690 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.075702+0000", 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.462558042, 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006662482, 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24690 2024-09-04T21:54:35.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075702+0000", 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075702+0000", 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075703+0000", 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075712+0000", 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2269999999999999e-06 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075725+0000", 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2262000000000001e-05 2024-09-04T21:54:35.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.075732+0000", 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0330000000000003e-06 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.077083+0000", 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001351323 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.077338+0000", 2024-09-04T21:54:35.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025505299999999999 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.078292+0000", 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00095375099999999997 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080448+0000", 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021567159999999999 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082012+0000", 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015639849999999999 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082182+0000", 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169863 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082365+0000", 2024-09-04T21:54:35.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018222600000000001 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24691 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.076136+0000", 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.462124196, 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0064383740000000002, 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.654 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24691 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076136+0000", 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076136+0000", 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076137+0000", 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.879999999999999e-07 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076143+0000", 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6949999999999997e-06 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076154+0000", 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1377e-05 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076161+0000", 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4819999999999997e-06 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.079334+0000", 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031723060000000002 2024-09-04T21:54:35.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.079434+0000", 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9853999999999996e-05 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.079840+0000", 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040624299999999998 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.081954+0000", 2024-09-04T21:54:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021146339999999998 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082386+0000", 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043110199999999999 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082506+0000", 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120565 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082574+0000", 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8127999999999996e-05 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24692 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.462024584, 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0065395399999999999, 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:35.660 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24692 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076236+0000", 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2699999999999999e-07 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076243+0000", 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8680000000000001e-06 2024-09-04T21:54:35.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076257+0000", 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3923999999999999e-05 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.076262+0000", 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9110000000000001e-06 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080801+0000", 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00453908 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.080893+0000", 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2e-05 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.081333+0000", 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044013599999999999 2024-09-04T21:54:35.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082612+0000", 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001279262 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082633+0000", 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0625e-05 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:35.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082711+0000", 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8319999999999996e-05 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:23.082775+0000", 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3687000000000002e-05 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:35.666 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:35.668 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.3 to 1 (10000) 2024-09-04T21:54:35.669 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:35.669 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:35.665+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:35.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-04T21:54:35.730 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone13 from 0xdedbdd9 (overlap 0x1680ce3) to 0x6c03dd 2024-09-04T21:54:35.769 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:35.766+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:35.869 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:35.866+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:35.884 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:35.884 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:35.884 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:35.884 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:35.898 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:54:35.910 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-09-04T21:54:35.971 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:35.966+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:36.014 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:36.014 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:36.015 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:36.015 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:36.015 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:36.028 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:54:36.071 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:36.066+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:36.153 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.5354184709999998, 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:36.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:54:36.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:36.158 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.5244132280000002, 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:54:36.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:54:36.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98073045400000003, 2024-09-04T21:54:36.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:54:36.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:54:36.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98070647799999999, 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:54:36.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:54:36.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98066579200000004, 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:36.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98063239300000005, 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.177 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:54:36.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98058912200000004, 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.182 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:54:36.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:36.186 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98054444100000004, 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:54:36.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98048328500000004, 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.191 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:54:36.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:54:36.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.195 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.980453244, 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.196 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:54:36.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.200 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.980408473, 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:54:36.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:54:36.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.980360181, 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.205 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:54:36.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98032453600000002, 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:54:36.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.214 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98007531999999997, 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:54:36.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.98003645100000003, 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.219 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.97488970600000002, 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:54:36.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:54:36.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.974846447, 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:54:36.228 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:54:36.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:54:36.232 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.97480257599999998, 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.233 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:54:36.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.237 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.97476629800000003, 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.241 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.97474019199999995, 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.242 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:54:36.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:54:36.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:54:36.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:54:36.246 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.246 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.246 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.246 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:36.246 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:36.246 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:36.247 INFO:teuthology.orchestra.run.smithi112.stdout:2918 write 0xec6045c thru 0xec62370 (0x1f15 bytes) 2024-09-04T21:54:36.247 INFO:teuthology.orchestra.run.smithi112.stdout:2919 writesame 0x24caf20 thru 0x24d1811 data_size 0xe(0x68f2 bytes) 2024-09-04T21:54:36.248 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:36.245+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:36.248 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:36.249 INFO:teuthology.orchestra.run.smithi112.stdout:2920 write 0xc993651 thru 0xc998620 (0x4fd0 bytes) 2024-09-04T21:54:36.256 INFO:teuthology.orchestra.run.smithi112.stdout:2921 read 0x2e484e5 thru 0x2e5539a (0xceb6 bytes) 2024-09-04T21:54:36.256 INFO:teuthology.orchestra.run.smithi112.stdout:2925 punch from 0x8eaf4c to 0x8f5e69, (0xaf1d bytes) 2024-09-04T21:54:36.256 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:36.256 INFO:teuthology.orchestra.run.smithi112.stdout:2930 clone 15 order 19 su 8192 sc 5 2024-09-04T21:54:36.349 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:36.346+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:36.370 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:36.370 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:36.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:36.473 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:36.470+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:36.550 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:36.549+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:36.551 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:36.551 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:36.551 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:36.551 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:36.551 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:36.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:36.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:54:36.627 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.4 to 1 (10000) 2024-09-04T21:54:36.651 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:36.649+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:36.663 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-04T21:54:36.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:36.738 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone14 from 0xec62371 (overlap 0xdedbdd9) to 0x42db8f4 2024-09-04T21:54:36.767 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090884165999999, 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090857380999999, 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090828448, 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.777 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:36.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090804801000001, 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.782 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:36.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:36.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090785019999998, 2024-09-04T21:54:36.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090765625, 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:36.791 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:36.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.795 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090746352, 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090727130000001, 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:36.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090708292999999, 2024-09-04T21:54:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090688889999999, 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090667834000001, 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:36.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090635656, 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.820 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:36.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090609435000001, 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:36.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090581226000001, 2024-09-04T21:54:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:36.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:36.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090561185999999, 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.834 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:36.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090541804000001, 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090518216, 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.843 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:36.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:36.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.847 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090497898999999, 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:36.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:36.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:36.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:36.852 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090478347000001, 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:36.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:36.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:36.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.090458588000001, 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:36.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:36.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:36.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:36.861 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:36.862 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:36.762+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:36.863 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:36.864 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:36.861+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:36.882 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:36.882 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:36.900 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:36.900 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:36.930 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:36.931 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:36.961 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-09-04T21:54:36.963 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:36.963 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:36.965 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:36.962+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:36.981 INFO:teuthology.orchestra.run.smithi112.stdout:2931 writesame 0x4e1e9c0 thru 0x4e24344 data_size 0x1(0x5985 bytes) 2024-09-04T21:54:37.000 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:37.001 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:37.007 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.007 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.011 INFO:teuthology.orchestra.run.smithi112.stdout:2932 read 0x90e48b8 thru 0x90f416a (0xf8b3 bytes) 2024-09-04T21:54:37.011 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:37.011 INFO:teuthology.orchestra.run.smithi112.stdout:2937 read 0x969e477 thru 0x96aba3e (0xd5c8 bytes) 2024-09-04T21:54:37.011 INFO:teuthology.orchestra.run.smithi112.stdout:2938 writesame 0x5dfb631 thru 0x5e01dc7 data_size 0x1(0x6797 bytes) 2024-09-04T21:54:37.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:37.036 INFO:teuthology.orchestra.run.smithi112.stdout:2940 read 0x91e0de5 thru 0x91ecc27 (0xbe43 bytes) 2024-09-04T21:54:37.036 INFO:teuthology.orchestra.run.smithi112.stdout:2942 write 0x1e51950 thru 0x1e604bb (0xeb6c bytes) 2024-09-04T21:54:37.038 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:37.038 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:37.053 INFO:teuthology.orchestra.run.smithi112.stdout:2943 write 0x68d7e3b thru 0x68d8fbd (0x1183 bytes) 2024-09-04T21:54:37.062 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:37.062 INFO:teuthology.orchestra.run.smithi112.stdout:2946 write 0xb186a5a thru 0xb18da3f (0x6fe6 bytes) 2024-09-04T21:54:37.066 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:37.062+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:37.072 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:37.072 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:37.078 INFO:teuthology.orchestra.run.smithi112.stdout:2947 write 0x69811a2 thru 0x6989d06 (0x8b65 bytes) 2024-09-04T21:54:37.093 INFO:teuthology.orchestra.run.smithi112.stdout:2949 write 0xc734ab thru 0xc76447 (0x2f9d bytes) 2024-09-04T21:54:37.096 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:37.096 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:37.149 INFO:teuthology.orchestra.run.smithi112.stdout:2951 writesame 0x28e1b90 thru 0x28f02ed data_size 0x5(0xe75e bytes) 2024-09-04T21:54:37.166 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:37.162+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:37.168 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:37.168 INFO:teuthology.orchestra.run.smithi112.stdout:2953 write 0x61e4ec5 thru 0x61ed55c (0x8698 bytes) 2024-09-04T21:54:37.172 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.172 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.183 INFO:teuthology.orchestra.run.smithi112.stdout:2954 writesame 0xb391faa thru 0xb39414d data_size 0x2(0x21a4 bytes) 2024-09-04T21:54:37.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:37.188 INFO:teuthology.orchestra.run.smithi112.stdout:2955 punch from 0x2c7c6d to 0x2c9ec2, (0x2255 bytes) 2024-09-04T21:54:37.188 INFO:teuthology.orchestra.run.smithi112.stdout:2956 writesame 0x6a627ad thru 0x6a6f95c data_size 0xb0(0xd1b0 bytes) 2024-09-04T21:54:37.192 INFO:teuthology.orchestra.run.smithi112.stdout:2957 read 0xe137aa3 thru 0xe14204a (0xa5a8 bytes) 2024-09-04T21:54:37.192 INFO:teuthology.orchestra.run.smithi112.stdout:2958 read 0xab4a1c9 thru 0xab5042a (0x6262 bytes) 2024-09-04T21:54:37.192 INFO:teuthology.orchestra.run.smithi112.stdout:2959 writesame 0xa8e209e thru 0xa8edb37 data_size 0xa(0xba9a bytes) 2024-09-04T21:54:37.203 INFO:teuthology.orchestra.run.smithi112.stdout:2960 write 0xd25a216 thru 0xd25c816 (0x2601 bytes) 2024-09-04T21:54:37.210 INFO:teuthology.orchestra.run.smithi112.stdout:2962 write 0x1cdae3 thru 0x1ce642 (0xb60 bytes) 2024-09-04T21:54:37.230 INFO:teuthology.orchestra.run.smithi112.stdout:2963 writesame 0xa319ec9 thru 0xa328801 data_size 0x1(0xe939 bytes) 2024-09-04T21:54:37.267 INFO:teuthology.orchestra.run.smithi112.stdout:2964 writesame 0xcd05ff1 thru 0xcd0ab5c data_size 0x6(0x4b6c bytes) 2024-09-04T21:54:37.267 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:37.262+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:37.281 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:37.281 INFO:teuthology.orchestra.run.smithi112.stdout:2970 read 0x7a5947c thru 0x7a606c0 (0x7245 bytes) 2024-09-04T21:54:37.281 INFO:teuthology.orchestra.run.smithi112.stdout:2972 write 0x29808b thru 0x29cda9 (0x4d1f bytes) 2024-09-04T21:54:37.319 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.319 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.335 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:37.347 INFO:teuthology.orchestra.run.smithi112.stdout:2973 trunc from 0xec62371 to 0x2765289 2024-09-04T21:54:37.368 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:37.365+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:37.419 INFO:teuthology.orchestra.run.smithi112.stdout:2974 writesame 0x7095ef2 thru 0x7098ef9 data_size 0x3a(0x3008 bytes) 2024-09-04T21:54:37.432 INFO:teuthology.orchestra.run.smithi112.stdout:2976 write 0x884d41a thru 0x8855291 (0x7e78 bytes) 2024-09-04T21:54:37.468 INFO:teuthology.orchestra.run.smithi112.stdout:2977 write 0xc922619 thru 0xc92366e (0x1056 bytes) 2024-09-04T21:54:37.468 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:37.465+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:37.471 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.471 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:37.489 INFO:teuthology.orchestra.run.smithi112.stdout:2978 write 0xdc381c4 thru 0xdc465ef (0xe42c bytes) 2024-09-04T21:54:37.515 INFO:teuthology.orchestra.run.smithi112.stdout:2979 trunc from 0xdc465f0 to 0x53101dc 2024-09-04T21:54:37.561 INFO:teuthology.orchestra.run.smithi112.stdout:2981 writesame 0xc927509 thru 0xc92c054 data_size 0x4f(0x4b4c bytes) 2024-09-04T21:54:37.561 INFO:teuthology.orchestra.run.smithi112.stdout:2982 write 0x4303f6b thru 0x43084fb (0x4591 bytes) 2024-09-04T21:54:37.569 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:37.566+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:37.573 INFO:teuthology.orchestra.run.smithi112.stdout:2983 read 0x37d2675 thru 0x37dfa82 (0xd40e bytes) 2024-09-04T21:54:37.573 INFO:teuthology.orchestra.run.smithi112.stdout:2984 writesame 0x8a25df thru 0x8a6589 data_size 0x3(0x3fab bytes) 2024-09-04T21:54:37.611 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.612 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:37.631 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:54:37.635 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.5 to 1 (10000) 2024-09-04T21:54:37.663 INFO:teuthology.orchestra.run.smithi112.stdout:2985 punch from 0xab324c0 to 0xab409bd, (0xe4fd bytes) 2024-09-04T21:54:37.663 INFO:teuthology.orchestra.run.smithi112.stdout:2989 read 0x57e94e7 thru 0x57f1b4f (0x8669 bytes) 2024-09-04T21:54:37.663 INFO:teuthology.orchestra.run.smithi112.stdout:2990 read 0x6c8a809 thru 0x6c8e7b9 (0x3fb1 bytes) 2024-09-04T21:54:37.663 INFO:teuthology.orchestra.run.smithi112.stdout:2991 read 0x54a30bb thru 0x54a6417 (0x335d bytes) 2024-09-04T21:54:37.663 INFO:teuthology.orchestra.run.smithi112.stdout:2992 write 0x5c2b590 thru 0x5c3418d (0x8bfe bytes) 2024-09-04T21:54:37.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-04T21:54:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:2994 punch from 0x282bcce to 0x2830054, (0x4386 bytes) 2024-09-04T21:54:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:2995 write 0x4c88197 thru 0x4c88e0c (0xc76 bytes) 2024-09-04T21:54:37.678 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:37.674+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:37.682 INFO:teuthology.orchestra.run.smithi112.stdout:2997 punch from 0xcb3c6 to 0xd0f05, (0x5b3f bytes) 2024-09-04T21:54:37.683 INFO:teuthology.orchestra.run.smithi112.stdout:2999 write 0x4f08e4e thru 0x4f0a03c (0x11ef bytes) 2024-09-04T21:54:37.697 INFO:teuthology.orchestra.run.smithi112.stdout:3001 trunc from 0xc92c055 to 0x68ca3a3 2024-09-04T21:54:37.719 INFO:teuthology.orchestra.run.smithi112.stdout:3002 write 0x741cc7f thru 0x741d381 (0x703 bytes) 2024-09-04T21:54:37.743 INFO:teuthology.orchestra.run.smithi112.stdout:3003 write 0xdb67aed thru 0xdb7339b (0xb8af bytes) 2024-09-04T21:54:37.757 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:37.757 INFO:teuthology.orchestra.run.smithi112.stdout:3006 read 0x8124dc thru 0x81c18c (0x9cb1 bytes) 2024-09-04T21:54:37.758 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.758 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.759 INFO:teuthology.orchestra.run.smithi112.stdout:3007 read 0x1be7d7f thru 0x1bf6134 (0xe3b6 bytes) 2024-09-04T21:54:37.759 INFO:teuthology.orchestra.run.smithi112.stdout:3009 read 0xa366a8e thru 0xa369ad0 (0x3043 bytes) 2024-09-04T21:54:37.771 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:37.766+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:37.772 INFO:teuthology.orchestra.run.smithi112.stdout:3010 trunc from 0xdb7339c to 0x2d34339 2024-09-04T21:54:37.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:37.826 INFO:teuthology.orchestra.run.smithi112.stdout:3011 write 0xebda622 thru 0xebe6bae (0xc58d bytes) 2024-09-04T21:54:37.843 INFO:teuthology.orchestra.run.smithi112.stdout:3012 write 0x8c41d01 thru 0x8c4d1e4 (0xb4e4 bytes) 2024-09-04T21:54:37.858 INFO:teuthology.orchestra.run.smithi112.stdout:3013 read 0x2792130 thru 0x27a115c (0xf02d bytes) 2024-09-04T21:54:37.859 INFO:teuthology.orchestra.run.smithi112.stdout:3015 write 0x9b1c009 thru 0x9b1c2e1 (0x2d9 bytes) 2024-09-04T21:54:37.865 INFO:teuthology.orchestra.run.smithi112.stdout:3016 read 0x4897f10 thru 0x489e93f (0x6a30 bytes) 2024-09-04T21:54:37.865 INFO:teuthology.orchestra.run.smithi112.stdout:3017 clone 16 order 20 su 8192 sc 8 2024-09-04T21:54:37.872 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:37.870+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:37.939 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:37.939 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:37.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:37.973 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:37.969+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:38.074 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:38.070+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:38.105 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:38.105 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:38.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:38.175 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:38.173+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:38.276 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:38.273+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:38.279 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:38.279 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:38.293 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:38.378 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:38.373+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:38.425 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:38.425 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:38.441 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:38.483 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:38.481+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:38.580 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:38.577+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:38.607 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:38.607 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:38.621 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:38.662 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.6 to 1 (10000) 2024-09-04T21:54:38.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-04T21:54:38.710 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:38.709+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:38.770 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone15 from 0xebe6baf (overlap 0x2765289) to 0x2374a48 2024-09-04T21:54:38.781 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:38.782+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:38.797 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:38.797 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:38.817 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:38.882 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:38.882+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:38.954 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-09-04T21:54:38.983 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:38.982+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:38.985 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:38.985 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:38.999 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:39.084 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:39.081+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:39.122 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:39.134 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:39.134 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:39.148 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:39.185 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:39.186+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:39.192 INFO:teuthology.orchestra.run.smithi112.stdout:3018 punch from 0x2f33739 to 0x2f3bc8b, (0x8552 bytes) 2024-09-04T21:54:39.199 INFO:teuthology.orchestra.run.smithi112.stdout:3020 write 0x1da01fa thru 0x1da64e7 (0x62ee bytes) 2024-09-04T21:54:39.227 INFO:teuthology.orchestra.run.smithi112.stdout:3024 write 0x431c3e3 thru 0x432028d (0x3eab bytes) 2024-09-04T21:54:39.233 INFO:teuthology.orchestra.run.smithi112.stdout:3026 read 0xa24383c thru 0xa24699b (0x3160 bytes) 2024-09-04T21:54:39.233 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:39.233 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:39.235 INFO:teuthology.orchestra.run.smithi112.stdout:3029 trunc from 0xebe6baf to 0xcaab223 2024-09-04T21:54:39.258 INFO:teuthology.orchestra.run.smithi112.stdout:3030 trunc from 0xcaab223 to 0x84166bc 2024-09-04T21:54:39.280 INFO:teuthology.orchestra.run.smithi112.stdout:3032 writesame 0x1df4d8f thru 0x1df6d1a data_size 0x6(0x1f8c bytes) 2024-09-04T21:54:39.283 INFO:teuthology.orchestra.run.smithi112.stdout:3033 read 0xb0004f thru 0xb05dd9 (0x5d8b bytes) 2024-09-04T21:54:39.284 INFO:teuthology.orchestra.run.smithi112.stdout:3034 punch from 0x7181177 to 0x7184896, (0x371f bytes) 2024-09-04T21:54:39.284 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:39.284 INFO:teuthology.orchestra.run.smithi112.stdout:3036 write 0x3d31c84 thru 0x3d3be89 (0xa206 bytes) 2024-09-04T21:54:39.285 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:39.285 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:39.286 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:39.281+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:39.298 INFO:teuthology.orchestra.run.smithi112.stdout:3038 trunc from 0x84166bc to 0x9a96c34 2024-09-04T21:54:39.304 INFO:teuthology.orchestra.run.smithi112.stdout:3039 writesame 0x8a467eb thru 0x8a4c988 data_size 0x69(0x619e bytes) 2024-09-04T21:54:39.304 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:39.305 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:39.322 INFO:teuthology.orchestra.run.smithi112.stdout:3040 trunc from 0x9a96c34 to 0xd519cf 2024-09-04T21:54:39.324 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:39.324 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:39.328 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:39.328 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:39.350 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:39.351 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:39.351 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:39.379 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:39.379 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:39.395 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:39.393+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:39.401 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:39.402 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:39.422 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:39.422 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:39.436 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:39.436 INFO:teuthology.orchestra.run.smithi112.stdout:3042 read 0x3df106 thru 0x3e2a17 (0x3912 bytes) 2024-09-04T21:54:39.436 INFO:teuthology.orchestra.run.smithi112.stdout:3043 punch from 0xb6649c to 0xb6e5f6, (0x815a bytes) 2024-09-04T21:54:39.443 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:39.444 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:39.447 INFO:teuthology.orchestra.run.smithi112.stdout:3046 writesame 0x391e6ac thru 0x392a456 data_size 0xf(0xbdab bytes) 2024-09-04T21:54:39.447 INFO:teuthology.orchestra.run.smithi112.stdout:3047 clone 17 order 20 su 16384 sc 10 2024-09-04T21:54:39.488 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:39.485+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:39.517 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:39.518 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:39.518 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:39.518 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:39.532 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:39.618 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:39.617+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:39.660 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:54:39.663 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.7 to 1 (10000) 2024-09-04T21:54:39.690 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:39.685+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:39.692 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-04T21:54:39.692 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:54:39.701 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:39.701 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:39.701 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:39.701 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:39.701 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:39.716 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:39.787 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone16 from 0x392a457 (overlap 0xd519cf) to 0xaa7b0c 2024-09-04T21:54:39.807 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:39.805+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:39.892 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:39.889+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:39.894 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14769 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.729363372000002, 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.008898853, 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.895 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14769 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160775+0000", 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160777+0000", 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.486e-06 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160786+0000", 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3740000000000004e-06 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160806+0000", 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0165000000000001e-05 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.160822+0000", 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6203000000000001e-05 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163870+0000", 2024-09-04T21:54:39.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003047812 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.164137+0000", 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000266643 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.164872+0000", 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073519099999999995 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167623+0000", 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027509129999999998 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169518+0000", 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001895274 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169593+0000", 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4270000000000006e-05 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169674+0000", 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1521999999999996e-05 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14770 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:39.901 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.729098630999999, 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0090832199999999995, 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14770 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161040+0000", 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161042+0000", 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4930000000000001e-06 2024-09-04T21:54:39.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161054+0000", 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1466e-05 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161079+0000", 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5114999999999999e-05 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.161097+0000", 2024-09-04T21:54:39.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8315999999999999e-05 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.165726+0000", 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0046287849999999998 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.165869+0000", 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014293300000000001 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.166506+0000", 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063653199999999998 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169697+0000", 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031909410000000001 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.169867+0000", 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170478 2024-09-04T21:54:39.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170053+0000", 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018593099999999999 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170123+0000", 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0229999999999997e-05 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:14771 3.10 3:0a5c83ea:::rbd_object_map.1188c13d44f0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.726805923000001, 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089354540000000007, 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 14771 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.908 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163333+0000", 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163335+0000", 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7030000000000002e-06 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163350+0000", 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4523e-05 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163387+0000", 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6817999999999999e-05 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.163408+0000", 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1715000000000001e-05 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167677+0000", 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042681560000000004 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.167880+0000", 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020378 2024-09-04T21:54:39.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.168602+0000", 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00072149099999999999 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.170348+0000", 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017465849999999999 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172076+0000", 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001727836 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172205+0000", 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012882700000000001 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:17.172268+0000", 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3020000000000003e-05 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.913 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:23212 3.0 e41/38 3:02a02ced:::rbd_data.118892c64562.0000000000000008:head v 41'114, mlcod=39'112)", 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.571780453999999, 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0092641379999999999, 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318358+0000", 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.318361+0000", 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.286e-06 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322409+0000", 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040474400000000002 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322451+0000", 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2240999999999998e-05 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322473+0000", 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1676e-05 2024-09-04T21:54:39.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322671+0000", 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019802099999999999 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.322713+0000", 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2645000000000003e-05 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.327478+0000", 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047643989999999999 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.327622+0000", 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014443 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.918 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:23214 3.1e e41/22 3:7a40d9fa:::rbd_data.118892c64562.0000000000000009:head v 41'75, mlcod=41'73)", 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.556957159, 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088767499999999992, 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333181+0000", 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.333184+0000", 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9469999999999999e-06 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335653+0000", 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024681690000000001 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335684+0000", 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1112000000000003e-05 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335697+0000", 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3329999999999999e-05 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335861+0000", 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016441900000000001 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.335929+0000", 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7953999999999993e-05 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.341970+0000", 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006040803 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.342058+0000", 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8016000000000006e-05 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:23926 3.a 3:547e81fb:::rbd_data.118892c64562.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,truncate 1622016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.141982103, 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010368176, 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.923 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 23926 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748156+0000", 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.748159+0000", 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8729999999999999e-06 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750340+0000", 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021802879999999998 2024-09-04T21:54:39.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750372+0000", 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2758e-05 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750395+0000", 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2368999999999999e-05 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750558+0000", 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016326000000000001 2024-09-04T21:54:39.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.750850+0000", 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000292117 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.751390+0000", 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054000999999999995 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.757506+0000", 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061158460000000003 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758289+0000", 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078254599999999996 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758407+0000", 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011886400000000001 2024-09-04T21:54:39.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:21.758525+0000", 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011724500000000001 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24315 3.a 3:5082ddb1:::rbd_object_map.1188c517ef04:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 17.276771793999998, 2024-09-04T21:54:39.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074709240000000003, 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24315 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613367+0000", 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613370+0000", 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.49e-06 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613385+0000", 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5112e-05 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613421+0000", 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5246000000000001e-05 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613438+0000", 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7051999999999999e-05 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613585+0000", 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014730799999999999 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.613612+0000", 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7212e-05 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.617673+0000", 2024-09-04T21:54:39.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040609959999999999 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.617898+0000", 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022498400000000001 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.618232+0000", 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033338999999999998 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.619944+0000", 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001712746 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620686+0000", 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074206200000000004 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620774+0000", 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7249000000000004e-05 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.620838+0000", 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4077000000000003e-05 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:24372 3.a 3:547e81fb:::rbd_data.118892c64562.0000000000000002:head [stat,truncate 1081344] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 17.207744447, 2024-09-04T21:54:39.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088444000000000005, 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 24372 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:39.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682394+0000", 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682398+0000", 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.924e-06 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682412+0000", 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3782e-05 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682458+0000", 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5738000000000002e-05 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682483+0000", 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5313e-05 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682668+0000", 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018540200000000001 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.682935+0000", 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000266412 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.683466+0000", 2024-09-04T21:54:39.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053094700000000004 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.689562+0000", 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060959769999999998 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.690972+0000", 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014100499999999999 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691102+0000", 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013006399999999999 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:22.691239+0000", 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013679100000000001 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.942 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:37992 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.827180+0000", 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0629582089999996, 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010907648000000001, 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 37992 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.827180+0000", 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.827180+0000", 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.827183+0000", 2024-09-04T21:54:39.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4339999999999999e-06 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.829949+0000", 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002766265 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.829967+0000", 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7960000000000001e-05 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.829975+0000", 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1939999999999998e-06 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830064+0000", 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8851999999999996e-05 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830129+0000", 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4760999999999999e-05 2024-09-04T21:54:39.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830353+0000", 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022383200000000001 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.835173+0000", 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048204570000000002 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.837989+0000", 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028157289999999999 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.838043+0000", 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4469000000000003e-05 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.838088+0000", 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4694999999999998e-05 2024-09-04T21:54:39.948 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:37993 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:head [delete] snapc 10=[10] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.830006+0000", 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0601325810000004, 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088175140000000003, 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.949 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 37993 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830006+0000", 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830006+0000", 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830007+0000", 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.466e-06 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830011+0000", 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4599999999999999e-06 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830020+0000", 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9660000000000002e-06 2024-09-04T21:54:39.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.830026+0000", 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7599999999999999e-06 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.833503+0000", 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034772380000000001 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.833606+0000", 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010325199999999999 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.833780+0000", 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000174296 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.835567+0000", 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017862450000000001 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.838746+0000", 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031797430000000001 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.838797+0000", 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1045999999999999e-05 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.838824+0000", 2024-09-04T21:54:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6041999999999999e-05 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38258 3.a 3:56f9609f:::rbd_data.11883c2bd208.0000000000000035:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,delete] snapc 11=[11] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:36.763185+0000", 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.1269538219999999, 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0080919200000000007, 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.955 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38258 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763185+0000", 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763185+0000", 2024-09-04T21:54:39.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763187+0000", 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5270000000000001e-06 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763200+0000", 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2746e-05 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763229+0000", 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9019000000000001e-05 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763249+0000", 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9996e-05 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763388+0000", 2024-09-04T21:54:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013849899999999999 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.763633+0000", 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024517300000000001 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.764250+0000", 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061700500000000001 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.766081+0000", 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018315060000000001 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771004+0000", 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0049230999999999997 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771150+0000", 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000145826 2024-09-04T21:54:39.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771277+0000", 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012652299999999999 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38273 3.10 3:0e030fe8:::rbd_data.11883c2bd208.0000000000000044:head [stat,delete] snapc 11=[11] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:36.771675+0000", 2024-09-04T21:54:39.961 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.1184634409999998, 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074950069999999997, 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38273 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771675+0000", 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771675+0000", 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771677+0000", 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9719999999999999e-06 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771686+0000", 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4200000000000007e-06 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771709+0000", 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3952999999999999e-05 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.771723+0000", 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3645000000000001e-05 2024-09-04T21:54:39.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.772114+0000", 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039091400000000002 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.774182+0000", 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020683889999999999 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.779170+0000", 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0049877139999999999 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38490 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:36.991528+0000", 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.898610562, 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0092612960000000005, 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.966 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38490 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.991528+0000", 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.991528+0000", 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.991537+0000", 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9560000000000003e-06 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.991547+0000", 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-05 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.991572+0000", 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4343000000000001e-05 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.991589+0000", 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7050000000000001e-05 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998074+0000", 2024-09-04T21:54:39.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00648476 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998215+0000", 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000140939 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998781+0000", 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00056632299999999998 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.000470+0000", 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001688784 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.000505+0000", 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5802e-05 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.000671+0000", 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165122 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.000789+0000", 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118717 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38493 3.a 3:5423b1d7:::rbd_data.11882296a3bb.000000000000009d:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:36.998128+0000", 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.8920108010000001, 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007511177, 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.972 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38493 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998128+0000", 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998128+0000", 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998131+0000", 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2100000000000002e-06 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998147+0000", 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6106000000000001e-05 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998170+0000", 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3176e-05 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998187+0000", 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7050000000000001e-05 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998350+0000", 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016236500000000001 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.998698+0000", 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034821299999999999 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:36.999568+0000", 2024-09-04T21:54:39.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086971699999999999 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.004892+0000", 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0053244859999999998 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.005337+0000", 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044487699999999999 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.005505+0000", 2024-09-04T21:54:39.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000167928 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.005639+0000", 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013404899999999999 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38497 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:37.014057+0000", 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.8760817469999997, 2024-09-04T21:54:39.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0071470290000000001, 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38497 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014057+0000", 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014057+0000", 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014058+0000", 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6330000000000001e-06 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014066+0000", 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4270000000000002e-06 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014080+0000", 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4178999999999999e-05 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014089+0000", 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2269999999999999e-06 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.017471+0000", 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033815630000000002 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.017726+0000", 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025549200000000001 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.018823+0000", 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001096475 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.020911+0000", 2024-09-04T21:54:39.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020881110000000001 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.020949+0000", 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8019000000000003e-05 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.021092+0000", 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000142686 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.021204+0000", 2024-09-04T21:54:39.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112217 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38498 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:37.014274+0000", 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.8758650399999999, 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0095862559999999996, 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38498 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014274+0000", 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014274+0000", 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014275+0000", 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0300000000000001e-06 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014280+0000", 2024-09-04T21:54:39.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4729999999999999e-06 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014292+0000", 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2422999999999999e-05 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.014301+0000", 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.021279+0000", 2024-09-04T21:54:39.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0069781770000000003 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.021424+0000", 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144239 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.022002+0000", 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057801700000000001 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.023455+0000", 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014531329999999999 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.023582+0000", 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012689600000000001 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.023753+0000", 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000171285 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.023860+0000", 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106777 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38504 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:37.038306+0000", 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.851832983, 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0090762369999999992, 2024-09-04T21:54:39.990 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38504 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038306+0000", 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038306+0000", 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038315+0000", 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0399999999999998e-06 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038326+0000", 2024-09-04T21:54:39.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1076e-05 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038341+0000", 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5383000000000002e-05 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038352+0000", 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.057e-05 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.041028+0000", 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026760769999999998 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.041177+0000", 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000148772 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.041752+0000", 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057594999999999999 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047148+0000", 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0053954329999999998 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047170+0000", 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1795999999999999e-05 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047250+0000", 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9970999999999997e-05 2024-09-04T21:54:39.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047382+0000", 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013216899999999999 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38505 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:37.038573+0000", 2024-09-04T21:54:39.996 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.8515652739999999, 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0093896580000000004, 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38505 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038573+0000", 2024-09-04T21:54:39.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038573+0000", 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038576+0000", 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5000000000000002e-06 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:39.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038588+0000", 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2223e-05 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038621+0000", 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2561999999999998e-05 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038636+0000", 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5823000000000001e-05 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:39.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.042833+0000", 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041970189999999997 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.042969+0000", 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135239 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.043544+0000", 2024-09-04T21:54:40.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057575300000000003 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047569+0000", 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040242610000000003 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047731+0000", 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162261 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047875+0000", 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144256 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047963+0000", 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7761000000000002e-05 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38506 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:40.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:37.038812+0000", 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.8513260740000002, 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0093432649999999999, 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38506 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038812+0000", 2024-09-04T21:54:40.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038812+0000", 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038814+0000", 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038822+0000", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0730000000000005e-06 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038837+0000", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5302999999999999e-05 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.038848+0000", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0307e-05 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.044571+0000", 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057236930000000002 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.044867+0000", 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000295984 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.045957+0000", 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001089528 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047711+0000", 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001753742 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.047985+0000", 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027439800000000002 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.048102+0000", 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117043 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.048156+0000", 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3588000000000003e-05 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:38507 3.10 3:0c69eaf9:::rbd_object_map.11882296a3bb:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:37.039016+0000", 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.85112298, 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.01177279, 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.008 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 38507 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.039016+0000", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.039016+0000", 2024-09-04T21:54:40.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.039017+0000", 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.46e-06 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.039024+0000", 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8229999999999997e-06 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.039046+0000", 2024-09-04T21:54:40.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2481999999999999e-05 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.039070+0000", 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3609e-05 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.048207+0000", 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0091366829999999996 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.048347+0000", 2024-09-04T21:54:40.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014039599999999999 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.048926+0000", 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057879600000000002 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.050409+0000", 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001483072 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.050456+0000", 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7305000000000002e-05 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.050652+0000", 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019585499999999999 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:37.050788+0000", 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013630900000000001 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.013 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.014 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.014 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.014 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.014 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:40.014 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-09-04T21:54:40.016 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:40.016 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:40.013+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:40.098 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:54:40.099 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:54:40.116 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:40.114+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:40.117 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-04T21:54:39.657131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542329948,"num_objects":532,"num_object_clones":138,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":165,"num_read":55708,"num_read_kb":320021,"num_write":7681,"num_write_kb":156620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":27405180,"num_keys_recovered":70,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5888,"ondisk_log_size":5888,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":262272,"kb_used_data":26304,"kb_used_omap":13,"kb_used_meta":235762,"kb_avail":660340608,"statfs":{"total":676457349120,"available":676188782592,"internally_reserved":0,"allocated":26935296,"data_stored":213856828,"data_compressed":10918519,"data_compressed_allocated":18530304,"data_compressed_original":209337043,"omap_allocated":13961,"internal_metadata":241420663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20237715,"num_objects":127,"num_object_clones":13,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-11,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":81,"num_read":7473,"num_read_kb":48566,"num_write":909,"num_write_kb":18329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1489032,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.041062"},"pg_stats":[{"pgid":"3.1f","version":"55'115","reported_seq":523,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658656+0000","last_change":"2024-09-04T21:53:57.363350+0000","last_active":"2024-09-04T21:54:38.658656+0000","last_peered":"2024-09-04T21:54:38.658656+0000","last_clean":"2024-09-04T21:54:38.658656+0000","last_became_active":"2024-09-04T21:53:55.347870+0000","last_became_peered":"2024-09-04T21:53:55.347870+0000","last_unstale":"2024-09-04T21:54:38.658656+0000","last_undegraded":"2024-09-04T21:54:38.658656+0000","last_fullsized":"2024-09-04T21:54:38.658656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:00:15.981618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003772475,"stat_sum":{"num_bytes":14582261,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":5,"num_read":881,"num_read_kb":1360,"num_write":111,"num_write_kb":751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"54'143","reported_seq":1567,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661945+0000","last_change":"2024-09-04T21:53:57.370393+0000","last_active":"2024-09-04T21:54:37.661945+0000","last_peered":"2024-09-04T21:54:37.661945+0000","last_clean":"2024-09-04T21:54:37.661945+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:54:37.661945+0000","last_undegraded":"2024-09-04T21:54:37.661945+0000","last_fullsized":"2024-09-04T21:54:37.661945+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:03:14.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372790000000002,"stat_sum":{"num_bytes":13573104,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":5,"num_read":1369,"num_read_kb":6420,"num_write":141,"num_write_kb":3771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"54'266","reported_seq":755,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.638117+0000","last_change":"2024-09-04T21:53:57.353806+0000","last_active":"2024-09-04T21:54:36.638117+0000","last_peered":"2024-09-04T21:54:36.638117+0000","last_clean":"2024-09-04T21:54:36.638117+0000","last_became_active":"2024-09-04T21:53:55.351712+0000","last_became_peered":"2024-09-04T21:53:55.351712+0000","last_unstale":"2024-09-04T21:54:36.638117+0000","last_undegraded":"2024-09-04T21:54:36.638117+0000","last_fullsized":"2024-09-04T21:54:36.638117+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:26:38.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00153718,"stat_sum":{"num_bytes":31195600,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":1270,"num_read_kb":3988,"num_write":435,"num_write_kb":2926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"54'118","reported_seq":1542,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.679595+0000","last_change":"2024-09-04T21:53:57.353548+0000","last_active":"2024-09-04T21:54:36.679595+0000","last_peered":"2024-09-04T21:54:36.679595+0000","last_clean":"2024-09-04T21:54:36.679595+0000","last_became_active":"2024-09-04T21:53:55.351512+0000","last_became_peered":"2024-09-04T21:53:55.351512+0000","last_unstale":"2024-09-04T21:54:36.679595+0000","last_undegraded":"2024-09-04T21:54:36.679595+0000","last_fullsized":"2024-09-04T21:54:36.679595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:24:08.699614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349371,"stat_sum":{"num_bytes":17323641,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":1385,"num_read_kb":18152,"num_write":113,"num_write_kb":2711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"55'90","reported_seq":789,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658976+0000","last_change":"2024-09-04T21:53:57.360592+0000","last_active":"2024-09-04T21:54:38.658976+0000","last_peered":"2024-09-04T21:54:38.658976+0000","last_clean":"2024-09-04T21:54:38.658976+0000","last_became_active":"2024-09-04T21:53:55.356638+0000","last_became_peered":"2024-09-04T21:53:55.356638+0000","last_unstale":"2024-09-04T21:54:38.658976+0000","last_undegraded":"2024-09-04T21:54:38.658976+0000","last_fullsized":"2024-09-04T21:54:38.658976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:08:58.396226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295954,"stat_sum":{"num_bytes":4937520,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":645,"num_read_kb":4993,"num_write":90,"num_write_kb":5221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"55'187","reported_seq":3055,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.713476+0000","last_change":"2024-09-04T21:54:13.554263+0000","last_active":"2024-09-04T21:54:38.713476+0000","last_peered":"2024-09-04T21:54:38.713476+0000","last_clean":"2024-09-04T21:54:38.713476+0000","last_became_active":"2024-09-04T21:54:13.481244+0000","last_became_peered":"2024-09-04T21:54:13.481244+0000","last_unstale":"2024-09-04T21:54:38.713476+0000","last_undegraded":"2024-09-04T21:54:38.713476+0000","last_fullsized":"2024-09-04T21:54:38.713476+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:13:51.307618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001385184,"stat_sum":{"num_bytes":27591260,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":2953,"num_read_kb":27610,"num_write":260,"num_write_kb":6433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":930020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"54'124","reported_seq":586,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661839+0000","last_change":"2024-09-04T21:54:13.510809+0000","last_active":"2024-09-04T21:54:37.661839+0000","last_peered":"2024-09-04T21:54:37.661839+0000","last_clean":"2024-09-04T21:54:37.661839+0000","last_became_active":"2024-09-04T21:54:13.478356+0000","last_became_peered":"2024-09-04T21:54:13.478356+0000","last_unstale":"2024-09-04T21:54:37.661839+0000","last_undegraded":"2024-09-04T21:54:37.661839+0000","last_fullsized":"2024-09-04T21:54:37.661839+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:28:48.511524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032359149999999998,"stat_sum":{"num_bytes":9376078,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":4,"num_read":393,"num_read_kb":10267,"num_write":167,"num_write_kb":3653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"54'204","reported_seq":2718,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661805+0000","last_change":"2024-09-04T21:53:57.368302+0000","last_active":"2024-09-04T21:54:37.661805+0000","last_peered":"2024-09-04T21:54:37.661805+0000","last_clean":"2024-09-04T21:54:37.661805+0000","last_became_active":"2024-09-04T21:53:55.349441+0000","last_became_peered":"2024-09-04T21:53:55.349441+0000","last_unstale":"2024-09-04T21:54:37.661805+0000","last_undegraded":"2024-09-04T21:54:37.661805+0000","last_fullsized":"2024-09-04T21:54:37.661805+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:51:43.637089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001549499,"stat_sum":{"num_bytes":21206265,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":4,"num_read":3597,"num_read_kb":23575,"num_write":230,"num_write_kb":6867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"55'631","reported_seq":1700,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.705658+0000","last_change":"2024-09-04T21:53:57.363273+0000","last_active":"2024-09-04T21:54:38.705658+0000","last_peered":"2024-09-04T21:54:38.705658+0000","last_clean":"2024-09-04T21:54:38.705658+0000","last_became_active":"2024-09-04T21:53:55.346553+0000","last_became_peered":"2024-09-04T21:53:55.346553+0000","last_unstale":"2024-09-04T21:54:38.705658+0000","last_undegraded":"2024-09-04T21:54:38.705658+0000","last_fullsized":"2024-09-04T21:54:38.705658+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:12:03.547056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035280929999999999,"stat_sum":{"num_bytes":29831839,"num_objects":26,"num_object_clones":2,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":3858,"num_read_kb":18509,"num_write":1001,"num_write_kb":4370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"53'77","reported_seq":591,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.637916+0000","last_change":"2024-09-04T21:54:31.703062+0000","last_active":"2024-09-04T21:54:36.637916+0000","last_peered":"2024-09-04T21:54:36.637916+0000","last_clean":"2024-09-04T21:54:36.637916+0000","last_became_active":"2024-09-04T21:54:31.586621+0000","last_became_peered":"2024-09-04T21:54:31.586621+0000","last_unstale":"2024-09-04T21:54:36.637916+0000","last_undegraded":"2024-09-04T21:54:36.637916+0000","last_fullsized":"2024-09-04T21:54:36.637916+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:18:27.799642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086423499999999998,"stat_sum":{"num_bytes":11732992,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":399,"num_read_kb":11184,"num_write":71,"num_write_kb":2783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"56'122","reported_seq":1134,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.406898+0000","last_change":"2024-09-04T21:54:31.683707+0000","last_active":"2024-09-04T21:54:39.406898+0000","last_peered":"2024-09-04T21:54:39.406898+0000","last_clean":"2024-09-04T21:54:39.406898+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:54:39.406898+0000","last_undegraded":"2024-09-04T21:54:39.406898+0000","last_fullsized":"2024-09-04T21:54:39.406898+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:35:54.586324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078592199999999999,"stat_sum":{"num_bytes":15758418,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":15,"num_read":1415,"num_read_kb":12858,"num_write":116,"num_write_kb":12562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"55'82","reported_seq":488,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665092+0000","last_change":"2024-09-04T21:54:31.673826+0000","last_active":"2024-09-04T21:54:38.665092+0000","last_peered":"2024-09-04T21:54:38.665092+0000","last_clean":"2024-09-04T21:54:38.665092+0000","last_became_active":"2024-09-04T21:54:31.582403+0000","last_became_peered":"2024-09-04T21:54:31.582403+0000","last_unstale":"2024-09-04T21:54:38.665092+0000","last_undegraded":"2024-09-04T21:54:38.665092+0000","last_fullsized":"2024-09-04T21:54:38.665092+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:03:31.143954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001035425,"stat_sum":{"num_bytes":824097,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":364,"num_read_kb":660,"num_write":108,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":529142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":99,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665691+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:54:38.665691+0000","last_peered":"2024-09-04T21:54:38.665691+0000","last_clean":"2024-09-04T21:54:38.665691+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:54:38.665691+0000","last_undegraded":"2024-09-04T21:54:38.665691+0000","last_fullsized":"2024-09-04T21:54:38.665691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"54'71","reported_seq":457,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658720+0000","last_change":"2024-09-04T21:54:13.551637+0000","last_active":"2024-09-04T21:54:38.658720+0000","last_peered":"2024-09-04T21:54:38.658720+0000","last_clean":"2024-09-04T21:54:38.658720+0000","last_became_active":"2024-09-04T21:54:13.481286+0000","last_became_peered":"2024-09-04T21:54:13.481286+0000","last_unstale":"2024-09-04T21:54:38.658720+0000","last_undegraded":"2024-09-04T21:54:38.658720+0000","last_fullsized":"2024-09-04T21:54:38.658720+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:39:21.751128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001390173,"stat_sum":{"num_bytes":4021866,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":3,"num_read":269,"num_read_kb":870,"num_write":72,"num_write_kb":1636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":90858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665596+0000","last_change":"2024-09-04T21:54:13.481230+0000","last_active":"2024-09-04T21:54:38.665596+0000","last_peered":"2024-09-04T21:54:38.665596+0000","last_clean":"2024-09-04T21:54:38.665596+0000","last_became_active":"2024-09-04T21:54:13.479390+0000","last_became_peered":"2024-09-04T21:54:13.479390+0000","last_unstale":"2024-09-04T21:54:38.665596+0000","last_undegraded":"2024-09-04T21:54:38.665596+0000","last_fullsized":"2024-09-04T21:54:38.665596+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:05.478515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"55'99","reported_seq":1465,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.662110+0000","last_change":"2024-09-04T21:54:31.773042+0000","last_active":"2024-09-04T21:54:37.662110+0000","last_peered":"2024-09-04T21:54:37.662110+0000","last_clean":"2024-09-04T21:54:37.662110+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:54:37.662110+0000","last_undegraded":"2024-09-04T21:54:37.662110+0000","last_fullsized":"2024-09-04T21:54:37.662110+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:09.417870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015346380000000001,"stat_sum":{"num_bytes":6248516,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":4,"num_read":1816,"num_read_kb":7984,"num_write":144,"num_write_kb":7061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.656429+0000","last_change":"2024-09-04T21:54:31.580192+0000","last_active":"2024-09-04T21:54:38.656429+0000","last_peered":"2024-09-04T21:54:38.656429+0000","last_clean":"2024-09-04T21:54:38.656429+0000","last_became_active":"2024-09-04T21:54:31.579813+0000","last_became_peered":"2024-09-04T21:54:31.579813+0000","last_unstale":"2024-09-04T21:54:38.656429+0000","last_undegraded":"2024-09-04T21:54:38.656429+0000","last_fullsized":"2024-09-04T21:54:38.656429+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:50:43.689052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"55'108","reported_seq":860,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.640062+0000","last_change":"2024-09-04T21:54:13.539266+0000","last_active":"2024-09-04T21:54:37.640062+0000","last_peered":"2024-09-04T21:54:37.640062+0000","last_clean":"2024-09-04T21:54:37.640062+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:54:37.640062+0000","last_undegraded":"2024-09-04T21:54:37.640062+0000","last_fullsized":"2024-09-04T21:54:37.640062+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:31:05.112490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091729799999999998,"stat_sum":{"num_bytes":5791793,"num_objects":14,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":1276,"num_read_kb":3486,"num_write":109,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658327+0000","last_change":"2024-09-04T21:54:32.583609+0000","last_active":"2024-09-04T21:54:38.658327+0000","last_peered":"2024-09-04T21:54:38.658327+0000","last_clean":"2024-09-04T21:54:38.658327+0000","last_became_active":"2024-09-04T21:54:32.583305+0000","last_became_peered":"2024-09-04T21:54:32.583305+0000","last_unstale":"2024-09-04T21:54:38.658327+0000","last_undegraded":"2024-09-04T21:54:38.658327+0000","last_fullsized":"2024-09-04T21:54:38.658327+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:53:05.591149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"53'238","reported_seq":784,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.638477+0000","last_change":"2024-09-04T21:53:57.353941+0000","last_active":"2024-09-04T21:54:36.638477+0000","last_peered":"2024-09-04T21:54:36.638477+0000","last_clean":"2024-09-04T21:54:36.638477+0000","last_became_active":"2024-09-04T21:53:55.344044+0000","last_became_peered":"2024-09-04T21:53:55.344044+0000","last_unstale":"2024-09-04T21:54:36.638477+0000","last_undegraded":"2024-09-04T21:54:36.638477+0000","last_fullsized":"2024-09-04T21:54:36.638477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:25:15.168279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012888800000000001,"stat_sum":{"num_bytes":21466407,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":1618,"num_read_kb":4175,"num_write":328,"num_write_kb":7725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"55'107","reported_seq":2092,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665168+0000","last_change":"2024-09-04T21:53:57.345952+0000","last_active":"2024-09-04T21:54:38.665168+0000","last_peered":"2024-09-04T21:54:38.665168+0000","last_clean":"2024-09-04T21:54:38.665168+0000","last_became_active":"2024-09-04T21:53:55.349651+0000","last_became_peered":"2024-09-04T21:53:55.349651+0000","last_unstale":"2024-09-04T21:54:38.665168+0000","last_undegraded":"2024-09-04T21:54:38.665168+0000","last_fullsized":"2024-09-04T21:54:38.665168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:50:52.606735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050067700000000004,"stat_sum":{"num_bytes":11807215,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1971,"num_read_kb":11171,"num_write":114,"num_write_kb":8998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":97,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.637552+0000","last_change":"2024-09-04T21:54:13.487884+0000","last_active":"2024-09-04T21:54:36.637552+0000","last_peered":"2024-09-04T21:54:36.637552+0000","last_clean":"2024-09-04T21:54:36.637552+0000","last_became_active":"2024-09-04T21:54:13.479904+0000","last_became_peered":"2024-09-04T21:54:13.479904+0000","last_unstale":"2024-09-04T21:54:36.637552+0000","last_undegraded":"2024-09-04T21:54:36.637552+0000","last_fullsized":"2024-09-04T21:54:36.637552+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:48:49.639164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"54'166","reported_seq":1619,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.639855+0000","last_change":"2024-09-04T21:54:16.342878+0000","last_active":"2024-09-04T21:54:37.639855+0000","last_peered":"2024-09-04T21:54:37.639855+0000","last_clean":"2024-09-04T21:54:37.639855+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:54:37.639855+0000","last_undegraded":"2024-09-04T21:54:37.639855+0000","last_fullsized":"2024-09-04T21:54:37.639855+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:07:15.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010876340000000001,"stat_sum":{"num_bytes":9550206,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":2384,"num_read_kb":5044,"num_write":153,"num_write_kb":3300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.639898+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:54:37.639898+0000","last_peered":"2024-09-04T21:54:37.639898+0000","last_clean":"2024-09-04T21:54:37.639898+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:54:37.639898+0000","last_undegraded":"2024-09-04T21:54:37.639898+0000","last_fullsized":"2024-09-04T21:54:37.639898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'281","reported_seq":2317,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658347+0000","last_change":"2024-09-04T21:53:57.360893+0000","last_active":"2024-09-04T21:54:38.658347+0000","last_peered":"2024-09-04T21:54:38.658347+0000","last_clean":"2024-09-04T21:54:38.658347+0000","last_became_active":"2024-09-04T21:53:55.354821+0000","last_became_peered":"2024-09-04T21:53:55.354821+0000","last_unstale":"2024-09-04T21:54:38.658347+0000","last_undegraded":"2024-09-04T21:54:38.658347+0000","last_fullsized":"2024-09-04T21:54:38.658347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:05.129174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396415,"stat_sum":{"num_bytes":38820598,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":4,"num_read":2964,"num_read_kb":22729,"num_write":434,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.655823+0000","last_change":"2024-09-04T21:54:32.582337+0000","last_active":"2024-09-04T21:54:38.655823+0000","last_peered":"2024-09-04T21:54:38.655823+0000","last_clean":"2024-09-04T21:54:38.655823+0000","last_became_active":"2024-09-04T21:54:32.581971+0000","last_became_peered":"2024-09-04T21:54:32.581971+0000","last_unstale":"2024-09-04T21:54:38.655823+0000","last_undegraded":"2024-09-04T21:54:38.655823+0000","last_fullsized":"2024-09-04T21:54:38.655823+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:17:41.830398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"54'190","reported_seq":2279,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661959+0000","last_change":"2024-09-04T21:53:57.369713+0000","last_active":"2024-09-04T21:54:37.661959+0000","last_peered":"2024-09-04T21:54:37.661959+0000","last_clean":"2024-09-04T21:54:37.661959+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:54:37.661959+0000","last_undegraded":"2024-09-04T21:54:37.661959+0000","last_fullsized":"2024-09-04T21:54:37.661959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:57:56.456158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026956770000000001,"stat_sum":{"num_bytes":24579898,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":4,"num_read":3028,"num_read_kb":10775,"num_write":183,"num_write_kb":2589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.637537+0000","last_change":"2024-09-04T21:53:47.813802+0000","last_active":"2024-09-04T21:54:36.637537+0000","last_peered":"2024-09-04T21:54:36.637537+0000","last_clean":"2024-09-04T21:54:36.637537+0000","last_became_active":"2024-09-04T21:53:45.789500+0000","last_became_peered":"2024-09-04T21:53:45.789500+0000","last_unstale":"2024-09-04T21:54:36.637537+0000","last_undegraded":"2024-09-04T21:54:36.637537+0000","last_fullsized":"2024-09-04T21:54:36.637537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:04:59.647236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":127,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.655730+0000","last_change":"2024-09-04T21:54:31.581065+0000","last_active":"2024-09-04T21:54:38.655730+0000","last_peered":"2024-09-04T21:54:38.655730+0000","last_clean":"2024-09-04T21:54:38.655730+0000","last_became_active":"2024-09-04T21:54:31.580475+0000","last_became_peered":"2024-09-04T21:54:31.580475+0000","last_unstale":"2024-09-04T21:54:38.655730+0000","last_undegraded":"2024-09-04T21:54:38.655730+0000","last_fullsized":"2024-09-04T21:54:38.655730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:01:22.413021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"55'364","reported_seq":2119,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661874+0000","last_change":"2024-09-04T21:54:34.607151+0000","last_active":"2024-09-04T21:54:37.661874+0000","last_peered":"2024-09-04T21:54:37.661874+0000","last_clean":"2024-09-04T21:54:37.661874+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:54:37.661874+0000","last_undegraded":"2024-09-04T21:54:37.661874+0000","last_fullsized":"2024-09-04T21:54:37.661874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:25:44.620503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001397155,"stat_sum":{"num_bytes":19617160,"num_objects":18,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":3333,"num_read_kb":7867,"num_write":506,"num_write_kb":4375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"55'91","reported_seq":481,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665153+0000","last_change":"2024-09-04T21:53:57.346105+0000","last_active":"2024-09-04T21:54:38.665153+0000","last_peered":"2024-09-04T21:54:38.665153+0000","last_clean":"2024-09-04T21:54:38.665153+0000","last_became_active":"2024-09-04T21:53:55.345842+0000","last_became_peered":"2024-09-04T21:53:55.345842+0000","last_unstale":"2024-09-04T21:54:38.665153+0000","last_undegraded":"2024-09-04T21:54:38.665153+0000","last_fullsized":"2024-09-04T21:54:38.665153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:14:35.493982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094585800000000005,"stat_sum":{"num_bytes":9733391,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":6,"num_read":313,"num_read_kb":2512,"num_write":87,"num_write_kb":3852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"55'295","reported_seq":1551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665612+0000","last_change":"2024-09-04T21:53:57.339901+0000","last_active":"2024-09-04T21:54:38.665612+0000","last_peered":"2024-09-04T21:54:38.665612+0000","last_clean":"2024-09-04T21:54:38.665612+0000","last_became_active":"2024-09-04T21:53:55.355393+0000","last_became_peered":"2024-09-04T21:53:55.355393+0000","last_unstale":"2024-09-04T21:54:38.665612+0000","last_undegraded":"2024-09-04T21:54:38.665612+0000","last_fullsized":"2024-09-04T21:54:38.665612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:48:26.467469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050858999999999995,"stat_sum":{"num_bytes":51964886,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":3,"num_read":2483,"num_read_kb":8263,"num_write":366,"num_write_kb":3825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"54'187","reported_seq":813,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661913+0000","last_change":"2024-09-04T21:53:57.369698+0000","last_active":"2024-09-04T21:54:37.661913+0000","last_peered":"2024-09-04T21:54:37.661913+0000","last_clean":"2024-09-04T21:54:37.661913+0000","last_became_active":"2024-09-04T21:53:55.349356+0000","last_became_peered":"2024-09-04T21:53:55.349356+0000","last_unstale":"2024-09-04T21:54:37.661913+0000","last_undegraded":"2024-09-04T21:54:37.661913+0000","last_fullsized":"2024-09-04T21:54:37.661913+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:39:09.863896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024267529999999998,"stat_sum":{"num_bytes":3997764,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1639,"num_read_kb":4248,"num_write":249,"num_write_kb":3636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"52'87","reported_seq":1261,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.638076+0000","last_change":"2024-09-04T21:53:57.353569+0000","last_active":"2024-09-04T21:54:36.638076+0000","last_peered":"2024-09-04T21:54:36.638076+0000","last_clean":"2024-09-04T21:54:36.638076+0000","last_became_active":"2024-09-04T21:53:55.344385+0000","last_became_peered":"2024-09-04T21:53:55.344385+0000","last_unstale":"2024-09-04T21:54:36.638076+0000","last_undegraded":"2024-09-04T21:54:36.638076+0000","last_fullsized":"2024-09-04T21:54:36.638076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:22:16.223221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361924,"stat_sum":{"num_bytes":31270327,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":1114,"num_read_kb":5804,"num_write":92,"num_write_kb":2333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"56'91","reported_seq":184,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.408625+0000","last_change":"2024-09-04T21:54:31.623898+0000","last_active":"2024-09-04T21:54:39.408625+0000","last_peered":"2024-09-04T21:54:39.408625+0000","last_clean":"2024-09-04T21:54:39.408625+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:54:39.408625+0000","last_undegraded":"2024-09-04T21:54:39.408625+0000","last_fullsized":"2024-09-04T21:54:39.408625+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:26:50.120780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091741999999999995,"stat_sum":{"num_bytes":5274653,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":7,"num_read":584,"num_read_kb":4562,"num_write":91,"num_write_kb":2991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"56'90","reported_seq":236,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.384007+0000","last_change":"2024-09-04T21:54:31.719167+0000","last_active":"2024-09-04T21:54:39.384007+0000","last_peered":"2024-09-04T21:54:39.384007+0000","last_clean":"2024-09-04T21:54:39.384007+0000","last_became_active":"2024-09-04T21:54:31.584135+0000","last_became_peered":"2024-09-04T21:54:31.584135+0000","last_unstale":"2024-09-04T21:54:39.384007+0000","last_undegraded":"2024-09-04T21:54:39.384007+0000","last_fullsized":"2024-09-04T21:54:39.384007+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:20:11.001133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066328299999999997,"stat_sum":{"num_bytes":8781395,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":7,"num_read":1268,"num_read_kb":7878,"num_write":132,"num_write_kb":3125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":894295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"56'163","reported_seq":706,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.294061+0000","last_change":"2024-09-04T21:54:34.612521+0000","last_active":"2024-09-04T21:54:39.294061+0000","last_peered":"2024-09-04T21:54:39.294061+0000","last_clean":"2024-09-04T21:54:39.294061+0000","last_became_active":"2024-09-04T21:54:34.608209+0000","last_became_peered":"2024-09-04T21:54:34.608209+0000","last_unstale":"2024-09-04T21:54:39.294061+0000","last_undegraded":"2024-09-04T21:54:39.294061+0000","last_fullsized":"2024-09-04T21:54:39.294061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:56:26.988086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025152799999999999,"stat_sum":{"num_bytes":6235225,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":1000,"num_read_kb":1257,"num_write":190,"num_write_kb":2185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":717304,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"56'259","reported_seq":879,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.666378+0000","last_change":"2024-09-04T21:53:57.339857+0000","last_active":"2024-09-04T21:54:38.666378+0000","last_peered":"2024-09-04T21:54:38.666378+0000","last_clean":"2024-09-04T21:54:38.666378+0000","last_became_active":"2024-09-04T21:53:55.352363+0000","last_became_peered":"2024-09-04T21:53:55.352363+0000","last_unstale":"2024-09-04T21:54:38.666378+0000","last_undegraded":"2024-09-04T21:54:38.666378+0000","last_fullsized":"2024-09-04T21:54:38.666378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:48:56.230765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000478915,"stat_sum":{"num_bytes":5660038,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":11,"num_read":1435,"num_read_kb":3102,"num_write":377,"num_write_kb":3669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"56'380","reported_seq":4232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.425866+0000","last_change":"2024-09-04T21:54:34.612100+0000","last_active":"2024-09-04T21:54:39.425866+0000","last_peered":"2024-09-04T21:54:39.425866+0000","last_clean":"2024-09-04T21:54:39.425866+0000","last_became_active":"2024-09-04T21:54:34.607984+0000","last_became_peered":"2024-09-04T21:54:34.607984+0000","last_unstale":"2024-09-04T21:54:39.425866+0000","last_undegraded":"2024-09-04T21:54:39.425866+0000","last_fullsized":"2024-09-04T21:54:39.425866+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:16:11.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023455609999999999,"stat_sum":{"num_bytes":29535335,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":5,"num_read":5474,"num_read_kb":37397,"num_write":535,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":6482454,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"55'94","reported_seq":1878,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658398+0000","last_change":"2024-09-04T21:54:31.647985+0000","last_active":"2024-09-04T21:54:38.658398+0000","last_peered":"2024-09-04T21:54:38.658398+0000","last_clean":"2024-09-04T21:54:38.658398+0000","last_became_active":"2024-09-04T21:54:31.581055+0000","last_became_peered":"2024-09-04T21:54:31.581055+0000","last_unstale":"2024-09-04T21:54:38.658398+0000","last_undegraded":"2024-09-04T21:54:38.658398+0000","last_fullsized":"2024-09-04T21:54:38.658398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:28:53.243613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010422490000000001,"stat_sum":{"num_bytes":33688597,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":1658,"num_read_kb":20112,"num_write":104,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":2112294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"56'293","reported_seq":891,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.382519+0000","last_change":"2024-09-04T21:54:34.609339+0000","last_active":"2024-09-04T21:54:39.382519+0000","last_peered":"2024-09-04T21:54:39.382519+0000","last_clean":"2024-09-04T21:54:39.382519+0000","last_became_active":"2024-09-04T21:54:34.607781+0000","last_became_peered":"2024-09-04T21:54:34.607781+0000","last_unstale":"2024-09-04T21:54:39.382519+0000","last_undegraded":"2024-09-04T21:54:39.382519+0000","last_fullsized":"2024-09-04T21:54:39.382519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:41:14.437741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091209699999999995,"stat_sum":{"num_bytes":15761216,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":6,"num_read":1486,"num_read_kb":11154,"num_write":469,"num_write_kb":3758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1583654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":541739561,"num_objects":528,"num_object_clones":138,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":165,"num_read":55642,"num_read_kb":319966,"num_write":7578,"num_write_kb":154990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":27405161,"num_keys_recovered":70,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22953984,"data_stored":213669543,"data_compressed":11102587,"data_compressed_allocated":18821120,"data_compressed_original":211038238,"omap_allocated":0,"internal_metadata":0},"log_size":5813,"ondisk_log_size":5813,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1892352,"data_stored":14025191,"data_compressed":560831,"data_compressed_allocated":1032192,"data_compressed_original":13617746,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32716,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339124,"statfs":{"total":96636764160,"available":96603262976,"internally_reserved":0,"allocated":5787648,"data_stored":43930293,"data_compressed":2596779,"data_compressed_allocated":4145152,"data_compressed_original":43014733,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39440,"kb_used_data":4096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332400,"statfs":{"total":96636764160,"available":96596377600,"internally_reserved":0,"allocated":4194304,"data_stored":31570328,"data_compressed":1870722,"data_compressed_allocated":2879488,"data_compressed_original":30821588,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39648,"kb_used_data":4296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332192,"statfs":{"total":96636764160,"available":96596164608,"internally_reserved":0,"allocated":4399104,"data_stored":40020005,"data_compressed":1758928,"data_compressed_allocated":3158016,"data_compressed_original":39312097,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1978368,"data_stored":14458932,"data_compressed":620732,"data_compressed_allocated":1142784,"data_compressed_original":14052695,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37560,"kb_used_data":5400,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94334280,"statfs":{"total":96636764160,"available":96598302720,"internally_reserved":0,"allocated":5529600,"data_stored":46607196,"data_compressed":2190167,"data_compressed_allocated":4157440,"data_compressed_original":45907968,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42588,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329252,"statfs":{"total":96636764160,"available":96593154048,"internally_reserved":0,"allocated":3153920,"data_stored":23244883,"data_compressed":1320360,"data_compressed_allocated":2015232,"data_compressed_original":22610216,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3510272,"data_stored":30700379,"data_compressed":1864798,"data_compressed_allocated":2842624,"data_compressed_original":30248148,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1376256,"data_stored":14191420,"data_compressed":620732,"data_compressed_allocated":1142784,"data_compressed_original":14052695,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3850240,"data_stored":40303289,"data_compressed":1762685,"data_compressed_allocated":3190784,"data_compressed_original":39836385,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5148672,"data_stored":43644565,"data_compressed":2596779,"data_compressed_allocated":4145152,"data_compressed_original":43014733,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":22977239,"data_compressed":1320360,"data_compressed_allocated":2015232,"data_compressed_original":22610216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":13745591,"data_compressed":560831,"data_compressed_allocated":1032192,"data_compressed_original":13617746,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4878336,"data_stored":46419204,"data_compressed":2184675,"data_compressed_allocated":4165632,"data_compressed_original":46022656,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:54:40.120 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:54:40.179 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:40.179 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:40.193 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:54:40.217 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:40.218+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:40.222 INFO:teuthology.orchestra.run.smithi112.stdout:3048 write 0xad4b732 thru 0xad51ac1 (0x6390 bytes) 2024-09-04T21:54:40.227 INFO:teuthology.orchestra.run.smithi112.stdout:3049 write 0x2fdfba5 thru 0x2fe5554 (0x59b0 bytes) 2024-09-04T21:54:40.233 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:40.234 INFO:teuthology.orchestra.run.smithi112.stdout:3051 trunc from 0xad51ac2 to 0xa5c0fb0 2024-09-04T21:54:40.247 INFO:teuthology.orchestra.run.smithi112.stdout:3054 punch from 0x688b0be to 0x68914c4, (0x6406 bytes) 2024-09-04T21:54:40.247 INFO:teuthology.orchestra.run.smithi112.stdout:3055 writesame 0x3e470cf thru 0x3e4cfbe data_size 0x10(0x5ef0 bytes) 2024-09-04T21:54:40.252 INFO:teuthology.orchestra.run.smithi112.stdout:3056 writesame 0x6ac35ae thru 0x6ad2ea8 data_size 0x1(0xf8fb bytes) 2024-09-04T21:54:40.296 INFO:teuthology.orchestra.run.smithi112.stdout:3057 trunc from 0xa5c0fb0 to 0x245b3a7 2024-09-04T21:54:40.317 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:40.318+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:40.324 INFO:teuthology.orchestra.run.smithi112.stdout:3058 punch from 0x17a9462 to 0x17b2855, (0x93f3 bytes) 2024-09-04T21:54:40.324 INFO:teuthology.orchestra.run.smithi112.stdout:3059 read 0x207eded thru 0x2085679 (0x688d bytes) 2024-09-04T21:54:40.325 INFO:teuthology.orchestra.run.smithi112.stdout:3060 read 0x1dca18e thru 0x1dcc380 (0x21f3 bytes) 2024-09-04T21:54:40.325 INFO:teuthology.orchestra.run.smithi112.stdout:3061 read 0xf31f35 thru 0xf400cc (0xe198 bytes) 2024-09-04T21:54:40.326 INFO:teuthology.orchestra.run.smithi112.stdout:3062 read 0xf5444 thru 0xf6e85 (0x1a42 bytes) 2024-09-04T21:54:40.334 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:40.334 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:40.334 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:40.334 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:40.334 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:40.339 INFO:teuthology.orchestra.run.smithi112.stdout:3063 write 0x74080aa thru 0x7417bd0 (0xfb27 bytes) 2024-09-04T21:54:40.351 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:54:40.358 INFO:teuthology.orchestra.run.smithi112.stdout:3064 write 0xc0ae913 thru 0xc0bcabf (0xe1ad bytes) 2024-09-04T21:54:40.370 INFO:teuthology.orchestra.run.smithi112.stdout:3068 trunc from 0xc0bcac0 to 0x754cf66 2024-09-04T21:54:40.383 INFO:teuthology.orchestra.run.smithi112.stdout:3069 read 0xaa24e0 thru 0xaa7e7d (0x599e bytes) 2024-09-04T21:54:40.384 INFO:teuthology.orchestra.run.smithi112.stdout:3070 writesame 0x433c050 thru 0x433f054 data_size 0x1(0x3005 bytes) 2024-09-04T21:54:40.395 INFO:teuthology.orchestra.run.smithi112.stdout:3071 read 0x66f1a12 thru 0x66feb05 (0xd0f4 bytes) 2024-09-04T21:54:40.400 INFO:teuthology.orchestra.run.smithi112.stdout:3072 trunc from 0x754cf66 to 0x3639f52 2024-09-04T21:54:40.412 INFO:teuthology.orchestra.run.smithi112.stdout:3074 punch from 0x4037ce to 0x40f351, (0xbb83 bytes) 2024-09-04T21:54:40.412 INFO:teuthology.orchestra.run.smithi112.stdout:3075 read 0x78e50b thru 0x791161 (0x2c57 bytes) 2024-09-04T21:54:40.412 INFO:teuthology.orchestra.run.smithi112.stdout:3076 read 0x86596 thru 0x8fc4a (0x96b5 bytes) 2024-09-04T21:54:40.417 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:40.418+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:40.418 INFO:teuthology.orchestra.run.smithi112.stdout:3077 trunc from 0x3639f52 to 0x7633e4 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8901803800000003, 2024-09-04T21:54:40.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:40.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:54:40.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:54:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:40.512 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8791751370000007, 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:54:40.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:40.517 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3354923630000002, 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:40.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:54:40.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:54:40.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.522 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3354683869999997, 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:54:40.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3354277010000004, 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:54:40.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3353943020000001, 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.532 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3353510310000001, 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.536 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:54:40.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3353063499999998, 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:54:40.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:54:40.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3352451939999996, 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:54:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:54:40.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:54:40.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.335215153, 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:54:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3351703820000003, 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.554 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.558 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3351220899999996, 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:54:40.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:40.563 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.335086445, 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:54:40.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3348372289999997, 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:54:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.572 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3347983599999997, 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.573 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:54:40.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3296516150000004, 2024-09-04T21:54:40.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:54:40.581 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3296083559999996, 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.582 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:40.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.586 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3295644849999997, 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3295282070000001, 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.592 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3295021010000001, 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:54:40.596 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:54:40.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:54:40.600 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.601 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:40.601 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:40.601 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:54:40.603 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:54:40.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:40.605 INFO:teuthology.orchestra.run.smithi112.stdout:3078 write 0xe7d2444 thru 0xe7def2d (0xcaea bytes) 2024-09-04T21:54:40.606 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-04T21:54:40.441327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542329948,"num_objects":532,"num_object_clones":138,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":165,"num_read":55708,"num_read_kb":320021,"num_write":7681,"num_write_kb":156620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":27405180,"num_keys_recovered":70,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5888,"ondisk_log_size":5888,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":262272,"kb_used_data":26304,"kb_used_omap":13,"kb_used_meta":235762,"kb_avail":660340608,"statfs":{"total":676457349120,"available":676188782592,"internally_reserved":0,"allocated":26935296,"data_stored":213856828,"data_compressed":10918519,"data_compressed_allocated":18530304,"data_compressed_original":209337043,"omap_allocated":13961,"internal_metadata":241420663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20237715,"num_objects":127,"num_object_clones":13,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-11,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":81,"num_read":7473,"num_read_kb":48566,"num_write":909,"num_write_kb":18329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1489032,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001325"},"pg_stats":[{"pgid":"3.1f","version":"55'115","reported_seq":523,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658656+0000","last_change":"2024-09-04T21:53:57.363350+0000","last_active":"2024-09-04T21:54:38.658656+0000","last_peered":"2024-09-04T21:54:38.658656+0000","last_clean":"2024-09-04T21:54:38.658656+0000","last_became_active":"2024-09-04T21:53:55.347870+0000","last_became_peered":"2024-09-04T21:53:55.347870+0000","last_unstale":"2024-09-04T21:54:38.658656+0000","last_undegraded":"2024-09-04T21:54:38.658656+0000","last_fullsized":"2024-09-04T21:54:38.658656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:00:15.981618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003772475,"stat_sum":{"num_bytes":14582261,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":5,"num_read":881,"num_read_kb":1360,"num_write":111,"num_write_kb":751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"54'143","reported_seq":1567,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661945+0000","last_change":"2024-09-04T21:53:57.370393+0000","last_active":"2024-09-04T21:54:37.661945+0000","last_peered":"2024-09-04T21:54:37.661945+0000","last_clean":"2024-09-04T21:54:37.661945+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:54:37.661945+0000","last_undegraded":"2024-09-04T21:54:37.661945+0000","last_fullsized":"2024-09-04T21:54:37.661945+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:03:14.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372790000000002,"stat_sum":{"num_bytes":13573104,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":5,"num_read":1369,"num_read_kb":6420,"num_write":141,"num_write_kb":3771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"54'266","reported_seq":755,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.638117+0000","last_change":"2024-09-04T21:53:57.353806+0000","last_active":"2024-09-04T21:54:36.638117+0000","last_peered":"2024-09-04T21:54:36.638117+0000","last_clean":"2024-09-04T21:54:36.638117+0000","last_became_active":"2024-09-04T21:53:55.351712+0000","last_became_peered":"2024-09-04T21:53:55.351712+0000","last_unstale":"2024-09-04T21:54:36.638117+0000","last_undegraded":"2024-09-04T21:54:36.638117+0000","last_fullsized":"2024-09-04T21:54:36.638117+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:26:38.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00153718,"stat_sum":{"num_bytes":31195600,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":1270,"num_read_kb":3988,"num_write":435,"num_write_kb":2926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"54'118","reported_seq":1542,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.679595+0000","last_change":"2024-09-04T21:53:57.353548+0000","last_active":"2024-09-04T21:54:36.679595+0000","last_peered":"2024-09-04T21:54:36.679595+0000","last_clean":"2024-09-04T21:54:36.679595+0000","last_became_active":"2024-09-04T21:53:55.351512+0000","last_became_peered":"2024-09-04T21:53:55.351512+0000","last_unstale":"2024-09-04T21:54:36.679595+0000","last_undegraded":"2024-09-04T21:54:36.679595+0000","last_fullsized":"2024-09-04T21:54:36.679595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:24:08.699614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349371,"stat_sum":{"num_bytes":17323641,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":1385,"num_read_kb":18152,"num_write":113,"num_write_kb":2711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"55'90","reported_seq":789,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658976+0000","last_change":"2024-09-04T21:53:57.360592+0000","last_active":"2024-09-04T21:54:38.658976+0000","last_peered":"2024-09-04T21:54:38.658976+0000","last_clean":"2024-09-04T21:54:38.658976+0000","last_became_active":"2024-09-04T21:53:55.356638+0000","last_became_peered":"2024-09-04T21:53:55.356638+0000","last_unstale":"2024-09-04T21:54:38.658976+0000","last_undegraded":"2024-09-04T21:54:38.658976+0000","last_fullsized":"2024-09-04T21:54:38.658976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:08:58.396226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295954,"stat_sum":{"num_bytes":4937520,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":645,"num_read_kb":4993,"num_write":90,"num_write_kb":5221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"55'187","reported_seq":3055,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.713476+0000","last_change":"2024-09-04T21:54:13.554263+0000","last_active":"2024-09-04T21:54:38.713476+0000","last_peered":"2024-09-04T21:54:38.713476+0000","last_clean":"2024-09-04T21:54:38.713476+0000","last_became_active":"2024-09-04T21:54:13.481244+0000","last_became_peered":"2024-09-04T21:54:13.481244+0000","last_unstale":"2024-09-04T21:54:38.713476+0000","last_undegraded":"2024-09-04T21:54:38.713476+0000","last_fullsized":"2024-09-04T21:54:38.713476+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:13:51.307618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001385184,"stat_sum":{"num_bytes":27591260,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":2953,"num_read_kb":27610,"num_write":260,"num_write_kb":6433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":930020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"54'124","reported_seq":586,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661839+0000","last_change":"2024-09-04T21:54:13.510809+0000","last_active":"2024-09-04T21:54:37.661839+0000","last_peered":"2024-09-04T21:54:37.661839+0000","last_clean":"2024-09-04T21:54:37.661839+0000","last_became_active":"2024-09-04T21:54:13.478356+0000","last_became_peered":"2024-09-04T21:54:13.478356+0000","last_unstale":"2024-09-04T21:54:37.661839+0000","last_undegraded":"2024-09-04T21:54:37.661839+0000","last_fullsized":"2024-09-04T21:54:37.661839+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:28:48.511524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032359149999999998,"stat_sum":{"num_bytes":9376078,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":4,"num_read":393,"num_read_kb":10267,"num_write":167,"num_write_kb":3653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"54'204","reported_seq":2718,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661805+0000","last_change":"2024-09-04T21:53:57.368302+0000","last_active":"2024-09-04T21:54:37.661805+0000","last_peered":"2024-09-04T21:54:37.661805+0000","last_clean":"2024-09-04T21:54:37.661805+0000","last_became_active":"2024-09-04T21:53:55.349441+0000","last_became_peered":"2024-09-04T21:53:55.349441+0000","last_unstale":"2024-09-04T21:54:37.661805+0000","last_undegraded":"2024-09-04T21:54:37.661805+0000","last_fullsized":"2024-09-04T21:54:37.661805+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:51:43.637089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001549499,"stat_sum":{"num_bytes":21206265,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":4,"num_read":3597,"num_read_kb":23575,"num_write":230,"num_write_kb":6867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"55'631","reported_seq":1700,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.705658+0000","last_change":"2024-09-04T21:53:57.363273+0000","last_active":"2024-09-04T21:54:38.705658+0000","last_peered":"2024-09-04T21:54:38.705658+0000","last_clean":"2024-09-04T21:54:38.705658+0000","last_became_active":"2024-09-04T21:53:55.346553+0000","last_became_peered":"2024-09-04T21:53:55.346553+0000","last_unstale":"2024-09-04T21:54:38.705658+0000","last_undegraded":"2024-09-04T21:54:38.705658+0000","last_fullsized":"2024-09-04T21:54:38.705658+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:12:03.547056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035280929999999999,"stat_sum":{"num_bytes":29831839,"num_objects":26,"num_object_clones":2,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":3858,"num_read_kb":18509,"num_write":1001,"num_write_kb":4370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"53'77","reported_seq":591,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.637916+0000","last_change":"2024-09-04T21:54:31.703062+0000","last_active":"2024-09-04T21:54:36.637916+0000","last_peered":"2024-09-04T21:54:36.637916+0000","last_clean":"2024-09-04T21:54:36.637916+0000","last_became_active":"2024-09-04T21:54:31.586621+0000","last_became_peered":"2024-09-04T21:54:31.586621+0000","last_unstale":"2024-09-04T21:54:36.637916+0000","last_undegraded":"2024-09-04T21:54:36.637916+0000","last_fullsized":"2024-09-04T21:54:36.637916+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:18:27.799642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086423499999999998,"stat_sum":{"num_bytes":11732992,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":399,"num_read_kb":11184,"num_write":71,"num_write_kb":2783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"56'122","reported_seq":1134,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.406898+0000","last_change":"2024-09-04T21:54:31.683707+0000","last_active":"2024-09-04T21:54:39.406898+0000","last_peered":"2024-09-04T21:54:39.406898+0000","last_clean":"2024-09-04T21:54:39.406898+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:54:39.406898+0000","last_undegraded":"2024-09-04T21:54:39.406898+0000","last_fullsized":"2024-09-04T21:54:39.406898+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:35:54.586324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078592199999999999,"stat_sum":{"num_bytes":15758418,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":15,"num_read":1415,"num_read_kb":12858,"num_write":116,"num_write_kb":12562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"55'82","reported_seq":488,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665092+0000","last_change":"2024-09-04T21:54:31.673826+0000","last_active":"2024-09-04T21:54:38.665092+0000","last_peered":"2024-09-04T21:54:38.665092+0000","last_clean":"2024-09-04T21:54:38.665092+0000","last_became_active":"2024-09-04T21:54:31.582403+0000","last_became_peered":"2024-09-04T21:54:31.582403+0000","last_unstale":"2024-09-04T21:54:38.665092+0000","last_undegraded":"2024-09-04T21:54:38.665092+0000","last_fullsized":"2024-09-04T21:54:38.665092+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:03:31.143954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001035425,"stat_sum":{"num_bytes":824097,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":364,"num_read_kb":660,"num_write":108,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":529142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":99,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665691+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:54:38.665691+0000","last_peered":"2024-09-04T21:54:38.665691+0000","last_clean":"2024-09-04T21:54:38.665691+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:54:38.665691+0000","last_undegraded":"2024-09-04T21:54:38.665691+0000","last_fullsized":"2024-09-04T21:54:38.665691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"54'71","reported_seq":457,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658720+0000","last_change":"2024-09-04T21:54:13.551637+0000","last_active":"2024-09-04T21:54:38.658720+0000","last_peered":"2024-09-04T21:54:38.658720+0000","last_clean":"2024-09-04T21:54:38.658720+0000","last_became_active":"2024-09-04T21:54:13.481286+0000","last_became_peered":"2024-09-04T21:54:13.481286+0000","last_unstale":"2024-09-04T21:54:38.658720+0000","last_undegraded":"2024-09-04T21:54:38.658720+0000","last_fullsized":"2024-09-04T21:54:38.658720+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:39:21.751128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001390173,"stat_sum":{"num_bytes":4021866,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":3,"num_read":269,"num_read_kb":870,"num_write":72,"num_write_kb":1636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":90858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665596+0000","last_change":"2024-09-04T21:54:13.481230+0000","last_active":"2024-09-04T21:54:38.665596+0000","last_peered":"2024-09-04T21:54:38.665596+0000","last_clean":"2024-09-04T21:54:38.665596+0000","last_became_active":"2024-09-04T21:54:13.479390+0000","last_became_peered":"2024-09-04T21:54:13.479390+0000","last_unstale":"2024-09-04T21:54:38.665596+0000","last_undegraded":"2024-09-04T21:54:38.665596+0000","last_fullsized":"2024-09-04T21:54:38.665596+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:05.478515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"55'99","reported_seq":1465,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.662110+0000","last_change":"2024-09-04T21:54:31.773042+0000","last_active":"2024-09-04T21:54:37.662110+0000","last_peered":"2024-09-04T21:54:37.662110+0000","last_clean":"2024-09-04T21:54:37.662110+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:54:37.662110+0000","last_undegraded":"2024-09-04T21:54:37.662110+0000","last_fullsized":"2024-09-04T21:54:37.662110+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:09.417870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015346380000000001,"stat_sum":{"num_bytes":6248516,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":4,"num_read":1816,"num_read_kb":7984,"num_write":144,"num_write_kb":7061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.656429+0000","last_change":"2024-09-04T21:54:31.580192+0000","last_active":"2024-09-04T21:54:38.656429+0000","last_peered":"2024-09-04T21:54:38.656429+0000","last_clean":"2024-09-04T21:54:38.656429+0000","last_became_active":"2024-09-04T21:54:31.579813+0000","last_became_peered":"2024-09-04T21:54:31.579813+0000","last_unstale":"2024-09-04T21:54:38.656429+0000","last_undegraded":"2024-09-04T21:54:38.656429+0000","last_fullsized":"2024-09-04T21:54:38.656429+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:50:43.689052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"55'108","reported_seq":860,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.640062+0000","last_change":"2024-09-04T21:54:13.539266+0000","last_active":"2024-09-04T21:54:37.640062+0000","last_peered":"2024-09-04T21:54:37.640062+0000","last_clean":"2024-09-04T21:54:37.640062+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:54:37.640062+0000","last_undegraded":"2024-09-04T21:54:37.640062+0000","last_fullsized":"2024-09-04T21:54:37.640062+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:31:05.112490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091729799999999998,"stat_sum":{"num_bytes":5791793,"num_objects":14,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":1276,"num_read_kb":3486,"num_write":109,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658327+0000","last_change":"2024-09-04T21:54:32.583609+0000","last_active":"2024-09-04T21:54:38.658327+0000","last_peered":"2024-09-04T21:54:38.658327+0000","last_clean":"2024-09-04T21:54:38.658327+0000","last_became_active":"2024-09-04T21:54:32.583305+0000","last_became_peered":"2024-09-04T21:54:32.583305+0000","last_unstale":"2024-09-04T21:54:38.658327+0000","last_undegraded":"2024-09-04T21:54:38.658327+0000","last_fullsized":"2024-09-04T21:54:38.658327+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:53:05.591149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"53'238","reported_seq":784,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.638477+0000","last_change":"2024-09-04T21:53:57.353941+0000","last_active":"2024-09-04T21:54:36.638477+0000","last_peered":"2024-09-04T21:54:36.638477+0000","last_clean":"2024-09-04T21:54:36.638477+0000","last_became_active":"2024-09-04T21:53:55.344044+0000","last_became_peered":"2024-09-04T21:53:55.344044+0000","last_unstale":"2024-09-04T21:54:36.638477+0000","last_undegraded":"2024-09-04T21:54:36.638477+0000","last_fullsized":"2024-09-04T21:54:36.638477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:25:15.168279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012888800000000001,"stat_sum":{"num_bytes":21466407,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":1618,"num_read_kb":4175,"num_write":328,"num_write_kb":7725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"55'107","reported_seq":2092,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665168+0000","last_change":"2024-09-04T21:53:57.345952+0000","last_active":"2024-09-04T21:54:38.665168+0000","last_peered":"2024-09-04T21:54:38.665168+0000","last_clean":"2024-09-04T21:54:38.665168+0000","last_became_active":"2024-09-04T21:53:55.349651+0000","last_became_peered":"2024-09-04T21:53:55.349651+0000","last_unstale":"2024-09-04T21:54:38.665168+0000","last_undegraded":"2024-09-04T21:54:38.665168+0000","last_fullsized":"2024-09-04T21:54:38.665168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:50:52.606735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050067700000000004,"stat_sum":{"num_bytes":11807215,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1971,"num_read_kb":11171,"num_write":114,"num_write_kb":8998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":97,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.637552+0000","last_change":"2024-09-04T21:54:13.487884+0000","last_active":"2024-09-04T21:54:36.637552+0000","last_peered":"2024-09-04T21:54:36.637552+0000","last_clean":"2024-09-04T21:54:36.637552+0000","last_became_active":"2024-09-04T21:54:13.479904+0000","last_became_peered":"2024-09-04T21:54:13.479904+0000","last_unstale":"2024-09-04T21:54:36.637552+0000","last_undegraded":"2024-09-04T21:54:36.637552+0000","last_fullsized":"2024-09-04T21:54:36.637552+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:48:49.639164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"54'166","reported_seq":1619,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.639855+0000","last_change":"2024-09-04T21:54:16.342878+0000","last_active":"2024-09-04T21:54:37.639855+0000","last_peered":"2024-09-04T21:54:37.639855+0000","last_clean":"2024-09-04T21:54:37.639855+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:54:37.639855+0000","last_undegraded":"2024-09-04T21:54:37.639855+0000","last_fullsized":"2024-09-04T21:54:37.639855+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:07:15.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010876340000000001,"stat_sum":{"num_bytes":9550206,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":2384,"num_read_kb":5044,"num_write":153,"num_write_kb":3300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.639898+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:54:37.639898+0000","last_peered":"2024-09-04T21:54:37.639898+0000","last_clean":"2024-09-04T21:54:37.639898+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:54:37.639898+0000","last_undegraded":"2024-09-04T21:54:37.639898+0000","last_fullsized":"2024-09-04T21:54:37.639898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"54'281","reported_seq":2317,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658347+0000","last_change":"2024-09-04T21:53:57.360893+0000","last_active":"2024-09-04T21:54:38.658347+0000","last_peered":"2024-09-04T21:54:38.658347+0000","last_clean":"2024-09-04T21:54:38.658347+0000","last_became_active":"2024-09-04T21:53:55.354821+0000","last_became_peered":"2024-09-04T21:53:55.354821+0000","last_unstale":"2024-09-04T21:54:38.658347+0000","last_undegraded":"2024-09-04T21:54:38.658347+0000","last_fullsized":"2024-09-04T21:54:38.658347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:05.129174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396415,"stat_sum":{"num_bytes":38820598,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":4,"num_read":2964,"num_read_kb":22729,"num_write":434,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.655823+0000","last_change":"2024-09-04T21:54:32.582337+0000","last_active":"2024-09-04T21:54:38.655823+0000","last_peered":"2024-09-04T21:54:38.655823+0000","last_clean":"2024-09-04T21:54:38.655823+0000","last_became_active":"2024-09-04T21:54:32.581971+0000","last_became_peered":"2024-09-04T21:54:32.581971+0000","last_unstale":"2024-09-04T21:54:38.655823+0000","last_undegraded":"2024-09-04T21:54:38.655823+0000","last_fullsized":"2024-09-04T21:54:38.655823+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:17:41.830398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"54'190","reported_seq":2279,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661959+0000","last_change":"2024-09-04T21:53:57.369713+0000","last_active":"2024-09-04T21:54:37.661959+0000","last_peered":"2024-09-04T21:54:37.661959+0000","last_clean":"2024-09-04T21:54:37.661959+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:54:37.661959+0000","last_undegraded":"2024-09-04T21:54:37.661959+0000","last_fullsized":"2024-09-04T21:54:37.661959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:57:56.456158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026956770000000001,"stat_sum":{"num_bytes":24579898,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":4,"num_read":3028,"num_read_kb":10775,"num_write":183,"num_write_kb":2589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.637537+0000","last_change":"2024-09-04T21:53:47.813802+0000","last_active":"2024-09-04T21:54:36.637537+0000","last_peered":"2024-09-04T21:54:36.637537+0000","last_clean":"2024-09-04T21:54:36.637537+0000","last_became_active":"2024-09-04T21:53:45.789500+0000","last_became_peered":"2024-09-04T21:53:45.789500+0000","last_unstale":"2024-09-04T21:54:36.637537+0000","last_undegraded":"2024-09-04T21:54:36.637537+0000","last_fullsized":"2024-09-04T21:54:36.637537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:04:59.647236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":127,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.655730+0000","last_change":"2024-09-04T21:54:31.581065+0000","last_active":"2024-09-04T21:54:38.655730+0000","last_peered":"2024-09-04T21:54:38.655730+0000","last_clean":"2024-09-04T21:54:38.655730+0000","last_became_active":"2024-09-04T21:54:31.580475+0000","last_became_peered":"2024-09-04T21:54:31.580475+0000","last_unstale":"2024-09-04T21:54:38.655730+0000","last_undegraded":"2024-09-04T21:54:38.655730+0000","last_fullsized":"2024-09-04T21:54:38.655730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:01:22.413021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"55'364","reported_seq":2119,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661874+0000","last_change":"2024-09-04T21:54:34.607151+0000","last_active":"2024-09-04T21:54:37.661874+0000","last_peered":"2024-09-04T21:54:37.661874+0000","last_clean":"2024-09-04T21:54:37.661874+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:54:37.661874+0000","last_undegraded":"2024-09-04T21:54:37.661874+0000","last_fullsized":"2024-09-04T21:54:37.661874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:25:44.620503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001397155,"stat_sum":{"num_bytes":19617160,"num_objects":18,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":3333,"num_read_kb":7867,"num_write":506,"num_write_kb":4375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"55'91","reported_seq":481,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665153+0000","last_change":"2024-09-04T21:53:57.346105+0000","last_active":"2024-09-04T21:54:38.665153+0000","last_peered":"2024-09-04T21:54:38.665153+0000","last_clean":"2024-09-04T21:54:38.665153+0000","last_became_active":"2024-09-04T21:53:55.345842+0000","last_became_peered":"2024-09-04T21:53:55.345842+0000","last_unstale":"2024-09-04T21:54:38.665153+0000","last_undegraded":"2024-09-04T21:54:38.665153+0000","last_fullsized":"2024-09-04T21:54:38.665153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:14:35.493982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094585800000000005,"stat_sum":{"num_bytes":9733391,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":6,"num_read":313,"num_read_kb":2512,"num_write":87,"num_write_kb":3852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"55'295","reported_seq":1551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.665612+0000","last_change":"2024-09-04T21:53:57.339901+0000","last_active":"2024-09-04T21:54:38.665612+0000","last_peered":"2024-09-04T21:54:38.665612+0000","last_clean":"2024-09-04T21:54:38.665612+0000","last_became_active":"2024-09-04T21:53:55.355393+0000","last_became_peered":"2024-09-04T21:53:55.355393+0000","last_unstale":"2024-09-04T21:54:38.665612+0000","last_undegraded":"2024-09-04T21:54:38.665612+0000","last_fullsized":"2024-09-04T21:54:38.665612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:48:26.467469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050858999999999995,"stat_sum":{"num_bytes":51964886,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":3,"num_read":2483,"num_read_kb":8263,"num_write":366,"num_write_kb":3825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"54'187","reported_seq":813,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-04T21:54:37.661913+0000","last_change":"2024-09-04T21:53:57.369698+0000","last_active":"2024-09-04T21:54:37.661913+0000","last_peered":"2024-09-04T21:54:37.661913+0000","last_clean":"2024-09-04T21:54:37.661913+0000","last_became_active":"2024-09-04T21:53:55.349356+0000","last_became_peered":"2024-09-04T21:53:55.349356+0000","last_unstale":"2024-09-04T21:54:37.661913+0000","last_undegraded":"2024-09-04T21:54:37.661913+0000","last_fullsized":"2024-09-04T21:54:37.661913+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:39:09.863896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024267529999999998,"stat_sum":{"num_bytes":3997764,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1639,"num_read_kb":4248,"num_write":249,"num_write_kb":3636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"52'87","reported_seq":1261,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-04T21:54:36.638076+0000","last_change":"2024-09-04T21:53:57.353569+0000","last_active":"2024-09-04T21:54:36.638076+0000","last_peered":"2024-09-04T21:54:36.638076+0000","last_clean":"2024-09-04T21:54:36.638076+0000","last_became_active":"2024-09-04T21:53:55.344385+0000","last_became_peered":"2024-09-04T21:53:55.344385+0000","last_unstale":"2024-09-04T21:54:36.638076+0000","last_undegraded":"2024-09-04T21:54:36.638076+0000","last_fullsized":"2024-09-04T21:54:36.638076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:22:16.223221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361924,"stat_sum":{"num_bytes":31270327,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":1114,"num_read_kb":5804,"num_write":92,"num_write_kb":2333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"56'91","reported_seq":184,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.408625+0000","last_change":"2024-09-04T21:54:31.623898+0000","last_active":"2024-09-04T21:54:39.408625+0000","last_peered":"2024-09-04T21:54:39.408625+0000","last_clean":"2024-09-04T21:54:39.408625+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:54:39.408625+0000","last_undegraded":"2024-09-04T21:54:39.408625+0000","last_fullsized":"2024-09-04T21:54:39.408625+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:26:50.120780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091741999999999995,"stat_sum":{"num_bytes":5274653,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":7,"num_read":584,"num_read_kb":4562,"num_write":91,"num_write_kb":2991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"56'90","reported_seq":236,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.384007+0000","last_change":"2024-09-04T21:54:31.719167+0000","last_active":"2024-09-04T21:54:39.384007+0000","last_peered":"2024-09-04T21:54:39.384007+0000","last_clean":"2024-09-04T21:54:39.384007+0000","last_became_active":"2024-09-04T21:54:31.584135+0000","last_became_peered":"2024-09-04T21:54:31.584135+0000","last_unstale":"2024-09-04T21:54:39.384007+0000","last_undegraded":"2024-09-04T21:54:39.384007+0000","last_fullsized":"2024-09-04T21:54:39.384007+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:20:11.001133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066328299999999997,"stat_sum":{"num_bytes":8781395,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":7,"num_read":1268,"num_read_kb":7878,"num_write":132,"num_write_kb":3125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":894295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"56'163","reported_seq":706,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.294061+0000","last_change":"2024-09-04T21:54:34.612521+0000","last_active":"2024-09-04T21:54:39.294061+0000","last_peered":"2024-09-04T21:54:39.294061+0000","last_clean":"2024-09-04T21:54:39.294061+0000","last_became_active":"2024-09-04T21:54:34.608209+0000","last_became_peered":"2024-09-04T21:54:34.608209+0000","last_unstale":"2024-09-04T21:54:39.294061+0000","last_undegraded":"2024-09-04T21:54:39.294061+0000","last_fullsized":"2024-09-04T21:54:39.294061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:56:26.988086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025152799999999999,"stat_sum":{"num_bytes":6235225,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":1000,"num_read_kb":1257,"num_write":190,"num_write_kb":2185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":717304,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"56'259","reported_seq":879,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.666378+0000","last_change":"2024-09-04T21:53:57.339857+0000","last_active":"2024-09-04T21:54:38.666378+0000","last_peered":"2024-09-04T21:54:38.666378+0000","last_clean":"2024-09-04T21:54:38.666378+0000","last_became_active":"2024-09-04T21:53:55.352363+0000","last_became_peered":"2024-09-04T21:53:55.352363+0000","last_unstale":"2024-09-04T21:54:38.666378+0000","last_undegraded":"2024-09-04T21:54:38.666378+0000","last_fullsized":"2024-09-04T21:54:38.666378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:48:56.230765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000478915,"stat_sum":{"num_bytes":5660038,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":11,"num_read":1435,"num_read_kb":3102,"num_write":377,"num_write_kb":3669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"56'380","reported_seq":4232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.425866+0000","last_change":"2024-09-04T21:54:34.612100+0000","last_active":"2024-09-04T21:54:39.425866+0000","last_peered":"2024-09-04T21:54:39.425866+0000","last_clean":"2024-09-04T21:54:39.425866+0000","last_became_active":"2024-09-04T21:54:34.607984+0000","last_became_peered":"2024-09-04T21:54:34.607984+0000","last_unstale":"2024-09-04T21:54:39.425866+0000","last_undegraded":"2024-09-04T21:54:39.425866+0000","last_fullsized":"2024-09-04T21:54:39.425866+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:16:11.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023455609999999999,"stat_sum":{"num_bytes":29535335,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":5,"num_read":5474,"num_read_kb":37397,"num_write":535,"num_write_kb":24380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":6482454,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"55'94","reported_seq":1878,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:38.658398+0000","last_change":"2024-09-04T21:54:31.647985+0000","last_active":"2024-09-04T21:54:38.658398+0000","last_peered":"2024-09-04T21:54:38.658398+0000","last_clean":"2024-09-04T21:54:38.658398+0000","last_became_active":"2024-09-04T21:54:31.581055+0000","last_became_peered":"2024-09-04T21:54:31.581055+0000","last_unstale":"2024-09-04T21:54:38.658398+0000","last_undegraded":"2024-09-04T21:54:38.658398+0000","last_fullsized":"2024-09-04T21:54:38.658398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:28:53.243613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010422490000000001,"stat_sum":{"num_bytes":33688597,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":1658,"num_read_kb":20112,"num_write":104,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":2112294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"56'293","reported_seq":891,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-04T21:54:39.382519+0000","last_change":"2024-09-04T21:54:34.609339+0000","last_active":"2024-09-04T21:54:39.382519+0000","last_peered":"2024-09-04T21:54:39.382519+0000","last_clean":"2024-09-04T21:54:39.382519+0000","last_became_active":"2024-09-04T21:54:34.607781+0000","last_became_peered":"2024-09-04T21:54:34.607781+0000","last_unstale":"2024-09-04T21:54:39.382519+0000","last_undegraded":"2024-09-04T21:54:39.382519+0000","last_fullsized":"2024-09-04T21:54:39.382519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:41:14.437741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091209699999999995,"stat_sum":{"num_bytes":15761216,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":6,"num_read":1486,"num_read_kb":11154,"num_write":469,"num_write_kb":3758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1583654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":541739561,"num_objects":528,"num_object_clones":138,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":165,"num_read":55642,"num_read_kb":319966,"num_write":7578,"num_write_kb":154990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":27405161,"num_keys_recovered":70,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22953984,"data_stored":213669543,"data_compressed":11102587,"data_compressed_allocated":18821120,"data_compressed_original":211038238,"omap_allocated":0,"internal_metadata":0},"log_size":5813,"ondisk_log_size":5813,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11848,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":1892352,"data_stored":14025191,"data_compressed":560831,"data_compressed_allocated":1032192,"data_compressed_original":13617746,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32716,"kb_used_data":5652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339124,"statfs":{"total":96636764160,"available":96603262976,"internally_reserved":0,"allocated":5787648,"data_stored":43930293,"data_compressed":2596779,"data_compressed_allocated":4145152,"data_compressed_original":43014733,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39440,"kb_used_data":4096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332400,"statfs":{"total":96636764160,"available":96596377600,"internally_reserved":0,"allocated":4194304,"data_stored":31570328,"data_compressed":1870722,"data_compressed_allocated":2879488,"data_compressed_original":30821588,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39648,"kb_used_data":4296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332192,"statfs":{"total":96636764160,"available":96596164608,"internally_reserved":0,"allocated":4399104,"data_stored":40020005,"data_compressed":1758928,"data_compressed_allocated":3158016,"data_compressed_original":39312097,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1978368,"data_stored":14458932,"data_compressed":620732,"data_compressed_allocated":1142784,"data_compressed_original":14052695,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37560,"kb_used_data":5400,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94334280,"statfs":{"total":96636764160,"available":96598302720,"internally_reserved":0,"allocated":5529600,"data_stored":46607196,"data_compressed":2190167,"data_compressed_allocated":4157440,"data_compressed_original":45907968,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42588,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329252,"statfs":{"total":96636764160,"available":96593154048,"internally_reserved":0,"allocated":3153920,"data_stored":23244883,"data_compressed":1320360,"data_compressed_allocated":2015232,"data_compressed_original":22610216,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3510272,"data_stored":30700379,"data_compressed":1864798,"data_compressed_allocated":2842624,"data_compressed_original":30248148,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1376256,"data_stored":14191420,"data_compressed":620732,"data_compressed_allocated":1142784,"data_compressed_original":14052695,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3850240,"data_stored":40303289,"data_compressed":1762685,"data_compressed_allocated":3190784,"data_compressed_original":39836385,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5148672,"data_stored":43644565,"data_compressed":2596779,"data_compressed_allocated":4145152,"data_compressed_original":43014733,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":22977239,"data_compressed":1320360,"data_compressed_allocated":2015232,"data_compressed_original":22610216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":13745591,"data_compressed":560831,"data_compressed_allocated":1032192,"data_compressed_original":13617746,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4878336,"data_stored":46419204,"data_compressed":2184675,"data_compressed_allocated":4165632,"data_compressed_original":46022656,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:54:40.608 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-04T21:54:40.608 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:54:40.608 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-04T21:54:40.608 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-09-04T21:54:40.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0 2024-09-04T21:54:40.609 INFO:teuthology.orchestra.run.smithi112.stdout:3081 punch from 0x823c58e to 0x8246a3b, (0xa4ad bytes) 2024-09-04T21:54:40.609 INFO:teuthology.orchestra.run.smithi112.stdout:3082 read 0xe75aad0 thru 0xe75f919 (0x4e4a bytes) 2024-09-04T21:54:40.609 INFO:teuthology.orchestra.run.smithi112.stdout:3083 trunc from 0xe7def2e to 0xd1ce5b 2024-09-04T21:54:40.610 INFO:teuthology.orchestra.run.smithi112.stdout:3084 write 0x2852311 thru 0x285f37a (0xd06a bytes) 2024-09-04T21:54:40.610 INFO:teuthology.orchestra.run.smithi112.stdout:3085 punch from 0x1afef75 to 0x1b00cb9, (0x1d44 bytes) 2024-09-04T21:54:40.610 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:40.602+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:40.612 INFO:teuthology.orchestra.run.smithi112.stdout:3086 writesame 0x7d59d67 thru 0x7d5d559 data_size 0x1(0x37f3 bytes) 2024-09-04T21:54:40.624 INFO:teuthology.orchestra.run.smithi112.stdout:3087 write 0xc267d17 thru 0xc269153 (0x143d bytes) 2024-09-04T21:54:40.629 INFO:teuthology.orchestra.run.smithi112.stdout:3088 read 0x838bc84 thru 0x83991b5 (0xd532 bytes) 2024-09-04T21:54:40.629 INFO:teuthology.orchestra.run.smithi112.stdout:3089 writesame 0x57765c1 thru 0x5781cb4 data_size 0x12(0xb6f4 bytes) 2024-09-04T21:54:40.650 INFO:teuthology.orchestra.run.smithi112.stdout:3090 trunc from 0xc269154 to 0x186eae8 2024-09-04T21:54:40.663 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:54:40.689 INFO:teuthology.orchestra.run.smithi112.stdout:3091 write 0x6b182b4 thru 0x6b1ac29 (0x2976 bytes) 2024-09-04T21:54:40.698 INFO:teuthology.orchestra.run.smithi112.stdout:3094 write 0x4025686 thru 0x402841c (0x2d97 bytes) 2024-09-04T21:54:40.703 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:40.702+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:40.705 INFO:teuthology.orchestra.run.smithi112.stdout:3095 read 0x5379623 thru 0x5381e23 (0x8801 bytes) 2024-09-04T21:54:40.705 INFO:teuthology.orchestra.run.smithi112.stdout:3096 read 0x43685bf thru 0x436c23b (0x3c7d bytes) 2024-09-04T21:54:40.705 INFO:teuthology.orchestra.run.smithi112.stdout:3099 write 0x5c7a51 thru 0x5d4e5f (0xd40f bytes) 2024-09-04T21:54:40.724 INFO:teuthology.orchestra.run.smithi112.stdout:3102 write 0x815efa1 thru 0x816a1ca (0xb22a bytes) 2024-09-04T21:54:40.741 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:40.741 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:40.741 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:40.742 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:40.745 INFO:teuthology.orchestra.run.smithi112.stdout:3103 write 0xb7d8635 thru 0xb7d8646 (0x12 bytes) 2024-09-04T21:54:40.751 INFO:teuthology.orchestra.run.smithi112.stdout:3107 read 0x12cd534 thru 0x12da408 (0xced5 bytes) 2024-09-04T21:54:40.751 INFO:teuthology.orchestra.run.smithi112.stdout:3108 read 0x240e14b thru 0x2413892 (0x5748 bytes) 2024-09-04T21:54:40.751 INFO:teuthology.orchestra.run.smithi112.stdout:3109 read 0xabb8105 thru 0xabc005f (0x7f5b bytes) 2024-09-04T21:54:40.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:40.763 INFO:teuthology.orchestra.run.smithi112.stdout:3110 write 0xe54099d thru 0xe543b60 (0x31c4 bytes) 2024-09-04T21:54:40.769 INFO:teuthology.orchestra.run.smithi112.stdout:3111 read 0xdf1894 thru 0xdfa57c (0x8ce9 bytes) 2024-09-04T21:54:40.769 INFO:teuthology.orchestra.run.smithi112.stdout:3112 write 0x4c6a25e thru 0x4c6c641 (0x23e4 bytes) 2024-09-04T21:54:40.775 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:40.775 INFO:teuthology.orchestra.run.smithi112.stdout:3114 punch from 0xe1d4089 to 0xe1dee98, (0xae0f bytes) 2024-09-04T21:54:40.775 INFO:teuthology.orchestra.run.smithi112.stdout:3115 punch from 0x763a9ac to 0x76466af, (0xbd03 bytes) 2024-09-04T21:54:40.775 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:40.775 INFO:teuthology.orchestra.run.smithi112.stdout:3117 punch from 0xc950560 to 0xc9533c4, (0x2e64 bytes) 2024-09-04T21:54:40.775 INFO:teuthology.orchestra.run.smithi112.stdout:3118 punch from 0x7a4a568 to 0x7a4eff7, (0x4a8f bytes) 2024-09-04T21:54:40.776 INFO:teuthology.orchestra.run.smithi112.stdout:3119 write 0xccfac25 thru 0xcd0a348 (0xf724 bytes) 2024-09-04T21:54:40.786 INFO:teuthology.orchestra.run.smithi112.stdout:3121 write 0x92d3104 thru 0x92dce4d (0x9d4a bytes) 2024-09-04T21:54:40.797 INFO:teuthology.orchestra.run.smithi112.stdout:3122 write 0xc8fd311 thru 0xc9008e4 (0x35d4 bytes) 2024-09-04T21:54:40.804 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:40.802+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:40.815 INFO:teuthology.orchestra.run.smithi112.stdout:3123 trunc from 0xe543b61 to 0x4407659 2024-09-04T21:54:40.847 INFO:teuthology.orchestra.run.smithi112.stdout:3125 punch from 0x10abb59 to 0x10b5553, (0x99fa bytes) 2024-09-04T21:54:40.872 INFO:teuthology.orchestra.run.smithi112.stdout:3126 writesame 0xdcb2fc6 thru 0xdcbc895 data_size 0x30(0x98d0 bytes) 2024-09-04T21:54:40.885 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:40.885 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:40.887 INFO:teuthology.orchestra.run.smithi112.stdout:3127 trunc from 0xdcbc896 to 0x2ae15a6 2024-09-04T21:54:40.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:40.911 INFO:teuthology.orchestra.run.smithi112.stdout:3130 punch from 0xa554a9 to 0xa60e91, (0xb9e8 bytes) 2024-09-04T21:54:40.911 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:40.910+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:40.924 INFO:teuthology.orchestra.run.smithi112.stdout:3131 trunc from 0x2ae15a6 to 0xc75f924 2024-09-04T21:54:40.949 INFO:teuthology.orchestra.run.smithi112.stdout:3132 write 0xe817bdf thru 0xe820b67 (0x8f89 bytes) 2024-09-04T21:54:40.958 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:40.958 INFO:teuthology.orchestra.run.smithi112.stdout:3134 read 0x5327b77 thru 0x532eba0 (0x702a bytes) 2024-09-04T21:54:40.959 INFO:teuthology.orchestra.run.smithi112.stdout:3136 read 0x495ce73 thru 0x49612e8 (0x4476 bytes) 2024-09-04T21:54:40.959 INFO:teuthology.orchestra.run.smithi112.stdout:3137 writesame 0x19b8a28 thru 0x19b9c42 data_size 0xf(0x121b bytes) 2024-09-04T21:54:40.964 INFO:teuthology.orchestra.run.smithi112.stdout:3138 read 0xf8cb09 thru 0xf8d200 (0x6f8 bytes) 2024-09-04T21:54:40.964 INFO:teuthology.orchestra.run.smithi112.stdout:3139 writesame 0x886399b thru 0x8869dfd data_size 0x1(0x6463 bytes) 2024-09-04T21:54:40.988 INFO:teuthology.orchestra.run.smithi112.stdout:3140 punch from 0x3df58d0 to 0x3e044bc, (0xebec bytes) 2024-09-04T21:54:40.989 INFO:teuthology.orchestra.run.smithi112.stdout:3141 write 0x1f3b380 thru 0x1f452f5 (0x9f76 bytes) 2024-09-04T21:54:40.997 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:40.997 INFO:teuthology.orchestra.run.smithi112.stdout:3143 write 0xbe73119 thru 0xbe80070 (0xcf58 bytes) 2024-09-04T21:54:41.006 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:41.006+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:41.006 INFO:teuthology.orchestra.run.smithi112.stdout:3144 writesame 0x83b8f4f thru 0x83bab60 data_size 0x1(0x1c12 bytes) 2024-09-04T21:54:41.011 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.011 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.011 INFO:teuthology.orchestra.run.smithi112.stdout:3151 write 0x3ba60bb thru 0x3bab60f (0x5555 bytes) 2024-09-04T21:54:41.018 INFO:teuthology.orchestra.run.smithi112.stdout:3152 write 0x6a24409 thru 0x6a29a04 (0x55fc bytes) 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358257606000002, 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:41.035 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:41.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358230820999999, 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.040 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:41.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358201888000004, 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.045 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:41.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:41.049 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358178240999997, 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:41.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358158459999999, 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.054 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:41.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358139065000003, 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:41.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:41.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:41.063 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358119791999997, 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:41.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:41.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358100569999998, 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.068 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.072 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358081732999999, 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:41.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358062330000003, 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:41.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:41.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358041274000001, 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.358009096000004, 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:41.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357982874999998, 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.090 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:41.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357954665999998, 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:41.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:41.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357934626000002, 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.099 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357915243999997, 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357891656, 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357871338999999, 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357851787000001, 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.357832027999997, 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:41.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.126 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:41.127 INFO:teuthology.orchestra.run.smithi112.stdout:3153 trunc from 0xe820b68 to 0x73e5c01 2024-09-04T21:54:41.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:41.129 INFO:teuthology.orchestra.run.smithi112.stdout:3154 punch from 0x281c792 to 0x281f703, (0x2f71 bytes) 2024-09-04T21:54:41.130 INFO:teuthology.orchestra.run.smithi112.stdout:3155 write 0xa48b53b thru 0xa48d685 (0x214b bytes) 2024-09-04T21:54:41.130 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.130 INFO:teuthology.orchestra.run.smithi112.stdout:3157 trunc from 0xa48d686 to 0xe2be4b2 2024-09-04T21:54:41.130 INFO:teuthology.orchestra.run.smithi112.stdout:3158 trunc from 0xe2be4b2 to 0x7375471 2024-09-04T21:54:41.130 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:41.126+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:41.146 INFO:teuthology.orchestra.run.smithi112.stdout:3159 writesame 0x6575161 thru 0x657ca27 data_size 0x1(0x78c7 bytes) 2024-09-04T21:54:41.177 INFO:teuthology.orchestra.run.smithi112.stdout:3160 read 0x72bad86 thru 0x72bdc67 (0x2ee2 bytes) 2024-09-04T21:54:41.182 INFO:teuthology.orchestra.run.smithi112.stdout:3162 trunc from 0x7375471 to 0x9d82e9d 2024-09-04T21:54:41.209 INFO:teuthology.orchestra.run.smithi112.stdout:3163 write 0xbb3f618 thru 0xbb4dd93 (0xe77c bytes) 2024-09-04T21:54:41.228 INFO:teuthology.orchestra.run.smithi112.stdout:3164 trunc from 0xbb4dd94 to 0x7ed8d46 2024-09-04T21:54:41.228 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:41.225+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:41.249 INFO:teuthology.orchestra.run.smithi112.stdout:3165 writesame 0x110fedd thru 0x11139d9 data_size 0x1(0x3afd bytes) 2024-09-04T21:54:41.265 INFO:teuthology.orchestra.run.smithi112.stdout:3166 trunc from 0x7ed8d46 to 0x79c8299 2024-09-04T21:54:41.277 INFO:teuthology.orchestra.run.smithi112.stdout:3167 read 0x16dd8d6 thru 0x16e067d (0x2da8 bytes) 2024-09-04T21:54:41.278 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.278 INFO:teuthology.orchestra.run.smithi112.stdout:3169 read 0x514a4a6 thru 0x515159c (0x70f7 bytes) 2024-09-04T21:54:41.278 INFO:teuthology.orchestra.run.smithi112.stdout:3170 read 0x3163be4 thru 0x31693ca (0x57e7 bytes) 2024-09-04T21:54:41.278 INFO:teuthology.orchestra.run.smithi112.stdout:3171 read 0x68c69ff thru 0x68cdfac (0x75ae bytes) 2024-09-04T21:54:41.278 INFO:teuthology.orchestra.run.smithi112.stdout:3172 read 0x69fe1eb thru 0x6a09772 (0xb588 bytes) 2024-09-04T21:54:41.279 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:41.279 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:41.287 INFO:teuthology.orchestra.run.smithi112.stdout:3173 trunc from 0x79c8299 to 0x4091d36 2024-09-04T21:54:41.296 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:41.311 INFO:teuthology.orchestra.run.smithi112.stdout:3175 punch from 0x4014dfd to 0x401fdd5, (0xafd8 bytes) 2024-09-04T21:54:41.311 INFO:teuthology.orchestra.run.smithi112.stdout:3176 read 0x114e6bf thru 0x1153199 (0x4adb bytes) 2024-09-04T21:54:41.312 INFO:teuthology.orchestra.run.smithi112.stdout:3177 read 0x33c5b88 thru 0x33cb074 (0x54ed bytes) 2024-09-04T21:54:41.312 INFO:teuthology.orchestra.run.smithi112.stdout:3178 punch from 0x619c5f to 0x6275ec, (0xd98d bytes) 2024-09-04T21:54:41.324 INFO:teuthology.orchestra.run.smithi112.stdout:3179 trunc from 0x4091d36 to 0xa677517 2024-09-04T21:54:41.329 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:41.330+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:41.337 INFO:teuthology.orchestra.run.smithi112.stdout:3182 punch from 0x9e24d04 to 0x9e2afb8, (0x62b4 bytes) 2024-09-04T21:54:41.337 INFO:teuthology.orchestra.run.smithi112.stdout:3184 write 0xa138938 thru 0xa144b3f (0xc208 bytes) 2024-09-04T21:54:41.347 INFO:teuthology.orchestra.run.smithi112.stdout:3185 read 0xbb8c5c thru 0xbc8718 (0xfabd bytes) 2024-09-04T21:54:41.348 INFO:teuthology.orchestra.run.smithi112.stdout:3186 read 0x6d3689a thru 0x6d41e60 (0xb5c7 bytes) 2024-09-04T21:54:41.348 INFO:teuthology.orchestra.run.smithi112.stdout:3187 punch from 0x3b24d18 to 0x3b27632, (0x291a bytes) 2024-09-04T21:54:41.348 INFO:teuthology.orchestra.run.smithi112.stdout:3188 trunc from 0xa677517 to 0xa6d1a43 2024-09-04T21:54:41.359 INFO:teuthology.orchestra.run.smithi112.stdout:3191 read 0x94a22e9 thru 0x94ab6ed (0x9405 bytes) 2024-09-04T21:54:41.359 INFO:teuthology.orchestra.run.smithi112.stdout:3193 writesame 0x2e212c thru 0x2f1c3a data_size 0x1(0xfb0f bytes) 2024-09-04T21:54:41.383 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.401 INFO:teuthology.orchestra.run.smithi112.stdout:3196 write 0xe5013bb thru 0xe5046fa (0x3340 bytes) 2024-09-04T21:54:41.425 INFO:teuthology.orchestra.run.smithi112.stdout:3197 trunc from 0xe5046fb to 0x2c45d7f 2024-09-04T21:54:41.432 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:41.429+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:41.442 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:41.442 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:41.442 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:41.442 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:41.442 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:41.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:41.462 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:41.462 INFO:teuthology.orchestra.run.smithi112.stdout:3199 write 0x2e033de thru 0x2e0ac53 (0x7876 bytes) 2024-09-04T21:54:41.471 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.491 INFO:teuthology.orchestra.run.smithi112.stdout:3201 write 0x7c5d562 thru 0x7c6404f (0x6aee bytes) 2024-09-04T21:54:41.503 INFO:teuthology.orchestra.run.smithi112.stdout:3203 trunc from 0x7c64050 to 0x9ca1444 2024-09-04T21:54:41.516 INFO:teuthology.orchestra.run.smithi112.stdout:3204 read 0x4d261c9 thru 0x4d26623 (0x45b bytes) 2024-09-04T21:54:41.526 INFO:teuthology.orchestra.run.smithi112.stdout:3205 trunc from 0x9ca1444 to 0x255d4b5 2024-09-04T21:54:41.531 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:41.530+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:41.551 INFO:teuthology.orchestra.run.smithi112.stdout:3206 read 0x7c2b73 thru 0x7cad11 (0x819f bytes) 2024-09-04T21:54:41.580 INFO:teuthology.orchestra.run.smithi112.stdout:3207 write 0xedc6112 thru 0xedcaa50 (0x493f bytes) 2024-09-04T21:54:41.586 INFO:teuthology.orchestra.run.smithi112.stdout:3208 write 0x52ee816 thru 0x52fe18c (0xf977 bytes) 2024-09-04T21:54:41.589 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:41.589 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.736990681000002, 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.590 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:41.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.595 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.610943417999998, 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.41438978, 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.387000282000002, 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:41.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.363735810999998, 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:41.612 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:41.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:41.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.335978021000003, 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:41.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:41.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.623 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.335862458999998, 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:41.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:41.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:41.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.629 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.035687674000002, 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.876555121999999, 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.635 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:41.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.876329063999997, 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:41.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:41.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.399115995999999, 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.652 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.169727885, 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.653 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:41.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.658 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.871537244000002, 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:41.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:41.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:41.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:41.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.871424388999998, 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:41.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:41.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:41.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.861763420999999, 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.674 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:41.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:41.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:41.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.841913695999999, 2024-09-04T21:54:41.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.682 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.69168501, 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:41.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:41.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:41.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.688 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.025358105000002, 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:41.689 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:41.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.022218918999997, 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:41.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.003907929, 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:41.703 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:41.704 INFO:teuthology.orchestra.run.smithi112.stdout:3209 write 0x65c30a2 thru 0x65c8bef (0x5b4e bytes) 2024-09-04T21:54:41.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:41.706 INFO:teuthology.orchestra.run.smithi112.stdout:3210 read 0xe865dc9 thru 0xe86b136 (0x536e bytes) 2024-09-04T21:54:41.706 INFO:teuthology.orchestra.run.smithi112.stdout:3212 write 0x8ffa9c2 thru 0x8fff3b6 (0x49f5 bytes) 2024-09-04T21:54:41.706 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:41.707 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:41.707 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:41.707 INFO:teuthology.orchestra.run.smithi112.stdout:3213 trunc from 0xedcaa51 to 0x9a885c 2024-09-04T21:54:41.707 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:41.708 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:41.708 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:41.708 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:41.708 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.5 primary-affinity to 0 (802) 2024-09-04T21:54:41.709 INFO:teuthology.orchestra.run.smithi112.stdout:3217 write 0xcac345d thru 0xcad28f0 (0xf494 bytes) 2024-09-04T21:54:41.709 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:41.709 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:41.710 INFO:teuthology.orchestra.run.smithi112.stdout:3218 trunc from 0xcad28f1 to 0x49544f7 2024-09-04T21:54:41.710 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:41.701+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:41.713 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:41.724 INFO:teuthology.orchestra.run.smithi112.stdout:3219 read 0x1855585 thru 0x18639b5 (0xe431 bytes) 2024-09-04T21:54:41.724 INFO:teuthology.orchestra.run.smithi112.stdout:3220 read 0x11fc292 thru 0x120b580 (0xf2ef bytes) 2024-09-04T21:54:41.724 INFO:teuthology.orchestra.run.smithi112.stdout:3221 read 0xfd0772 thru 0xfdd258 (0xcae7 bytes) 2024-09-04T21:54:41.724 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:41.724 INFO:teuthology.orchestra.run.smithi112.stdout:3224 writesame 0x1ba9cc1 thru 0x1bb2b54 data_size 0x7d(0x8e94 bytes) 2024-09-04T21:54:41.736 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:41.756 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:41.756 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:41.820 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:41.822+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:41.824 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:41.824 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:41.837 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:41.925 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:41.926+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:41.996 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:41.996 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.007 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:42.006+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:42.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:42.109 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:42.105+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:42.170 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:42.170 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.184 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:42.229 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:42.230+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:42.308 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:42.309 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.310 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:42.309+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:42.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:42.411 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:42.409+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:42.490 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:42.490 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.504 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:42.553 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:42.554+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:42.613 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:42.614+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:42.628 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:42.628 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:42.685 INFO:teuthology.orchestra.run.smithi112.stdout:3225 write 0x461e6f2 thru 0x46226d8 (0x3fe7 bytes) 2024-09-04T21:54:42.693 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:54:42.694 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:42.710 INFO:teuthology.orchestra.run.smithi112.stdout:3227 writesame 0x9357150 thru 0x9363dfd data_size 0x8e(0xccae bytes) 2024-09-04T21:54:42.711 INFO:teuthology.orchestra.run.smithi112.stdout:3228 trunc from 0x9363dfe to 0x85b706a 2024-09-04T21:54:42.713 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:42.714+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:42.728 INFO:teuthology.orchestra.run.smithi112.stdout:3231 trunc from 0x85b706a to 0xc7eb830 2024-09-04T21:54:42.738 INFO:teuthology.orchestra.run.smithi112.stdout:3232 read 0x8816c1a thru 0x881e248 (0x762f bytes) 2024-09-04T21:54:42.738 INFO:teuthology.orchestra.run.smithi112.stdout:3233 write 0x80ba727 thru 0x80c80df (0xd9b9 bytes) 2024-09-04T21:54:42.750 INFO:teuthology.orchestra.run.smithi112.stdout:3234 write 0x4dac35e thru 0x4dba7ca (0xe46d bytes) 2024-09-04T21:54:42.759 INFO:teuthology.orchestra.run.smithi112.stdout:3235 punch from 0xa09d1d1 to 0xa0a7b13, (0xa942 bytes) 2024-09-04T21:54:42.759 INFO:teuthology.orchestra.run.smithi112.stdout:3236 write 0x4979f9d thru 0x4985d7e (0xbde2 bytes) 2024-09-04T21:54:42.766 INFO:teuthology.orchestra.run.smithi112.stdout:3237 punch from 0x7c9e1c2 to 0x7ca2e47, (0x4c85 bytes) 2024-09-04T21:54:42.768 INFO:teuthology.orchestra.run.smithi112.stdout:3238 trunc from 0xc7eb830 to 0xd9b9466 2024-09-04T21:54:42.779 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:42.779 INFO:teuthology.orchestra.run.smithi112.stdout:3240 writesame 0x6e4af48 thru 0x6e56048 data_size 0x1(0xb101 bytes) 2024-09-04T21:54:42.802 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:42.802 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.804 INFO:teuthology.orchestra.run.smithi112.stdout:3241 write 0x4d92a61 thru 0x4d9475e (0x1cfe bytes) 2024-09-04T21:54:42.807 INFO:teuthology.orchestra.run.smithi112.stdout:3242 read 0xadd730f thru 0xade6c2f (0xf921 bytes) 2024-09-04T21:54:42.807 INFO:teuthology.orchestra.run.smithi112.stdout:3245 read 0xa78d92a thru 0xa78e8a9 (0xf80 bytes) 2024-09-04T21:54:42.808 INFO:teuthology.orchestra.run.smithi112.stdout:3246 write 0x3ba45f thru 0x3c2a80 (0x8622 bytes) 2024-09-04T21:54:42.811 INFO:teuthology.orchestra.run.smithi112.stdout:3247 writesame 0xc9cd974 thru 0xc9d5bd9 data_size 0x1(0x8266 bytes) 2024-09-04T21:54:42.814 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:42.814+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:42.818 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:42.832 INFO:teuthology.orchestra.run.smithi112.stdout:3248 read 0xc271dd1 thru 0xc273c81 (0x1eb1 bytes) 2024-09-04T21:54:42.838 INFO:teuthology.orchestra.run.smithi112.stdout:3249 trunc from 0xd9b9466 to 0xa3249a0 2024-09-04T21:54:42.849 INFO:teuthology.orchestra.run.smithi112.stdout:3250 read 0x8a74bc thru 0x8ac45b (0x4fa0 bytes) 2024-09-04T21:54:42.861 INFO:teuthology.orchestra.run.smithi112.stdout:3251 write 0xd2adc77 thru 0xd2bc18d (0xe517 bytes) 2024-09-04T21:54:42.867 INFO:teuthology.orchestra.run.smithi112.stdout:3252 punch from 0x507f819 to 0x508bc02, (0xc3e9 bytes) 2024-09-04T21:54:42.876 INFO:teuthology.orchestra.run.smithi112.stdout:3254 trunc from 0xd2bc18e to 0x71f471e 2024-09-04T21:54:42.896 INFO:teuthology.orchestra.run.smithi112.stdout:3255 trunc from 0x71f471e to 0xacfb11f 2024-09-04T21:54:42.903 INFO:teuthology.orchestra.run.smithi112.stdout:3256 write 0x7254afb thru 0x725c9b7 (0x7ebd bytes) 2024-09-04T21:54:42.915 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:42.914+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:42.920 INFO:teuthology.orchestra.run.smithi112.stdout:3257 write 0xecb6331 thru 0xecc365c (0xd32c bytes) 2024-09-04T21:54:42.927 INFO:teuthology.orchestra.run.smithi112.stdout:3259 read 0x2ecf77d thru 0x2ed8829 (0x90ad bytes) 2024-09-04T21:54:42.927 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:42.928 INFO:teuthology.orchestra.run.smithi112.stdout:3261 writesame 0x1e01889 thru 0x1e09eef data_size 0x3(0x8667 bytes) 2024-09-04T21:54:42.937 INFO:teuthology.orchestra.run.smithi112.stdout:3262 write 0xd942fae thru 0xd952f74 (0xffc7 bytes) 2024-09-04T21:54:42.947 INFO:teuthology.orchestra.run.smithi112.stdout:3263 trunc from 0xecc365d to 0xedd37a8 2024-09-04T21:54:42.953 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:42.953 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:42.961 INFO:teuthology.orchestra.run.smithi112.stdout:3264 trunc from 0xedd37a8 to 0xac570ee 2024-09-04T21:54:42.968 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:42.978 INFO:teuthology.orchestra.run.smithi112.stdout:3265 write 0x9430d02 thru 0x943f97f (0xec7e bytes) 2024-09-04T21:54:42.994 INFO:teuthology.orchestra.run.smithi112.stdout:3266 writesame 0xb1c0b80 thru 0xb1c4ac6 data_size 0x1(0x3f47 bytes) 2024-09-04T21:54:42.994 INFO:teuthology.orchestra.run.smithi112.stdout:3270 read 0x765164f thru 0x7655949 (0x42fb bytes) 2024-09-04T21:54:43.009 INFO:teuthology.orchestra.run.smithi112.stdout:3271 trunc from 0xb1c4ac7 to 0x5d2d45 2024-09-04T21:54:43.016 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:43.013+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:43.034 INFO:teuthology.orchestra.run.smithi112.stdout:3272 read 0xb9c98 thru 0xc29f3 (0x8d5c bytes) 2024-09-04T21:54:43.035 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.035 INFO:teuthology.orchestra.run.smithi112.stdout:3275 punch from 0x4fcb57 to 0x506ee1, (0xa38a bytes) 2024-09-04T21:54:43.035 INFO:teuthology.orchestra.run.smithi112.stdout:3276 read 0x179662 thru 0x1819da (0x8379 bytes) 2024-09-04T21:54:43.035 INFO:teuthology.orchestra.run.smithi112.stdout:3277 read 0x4ce7b4 thru 0x4d42c8 (0x5b15 bytes) 2024-09-04T21:54:43.064 INFO:teuthology.orchestra.run.smithi112.stdout:3279 writesame 0xe42cb1a thru 0xe431f8e data_size 0x1(0x5475 bytes) 2024-09-04T21:54:43.064 INFO:teuthology.orchestra.run.smithi112.stdout:3280 read 0x4996869 thru 0x499b427 (0x4bbf bytes) 2024-09-04T21:54:43.064 INFO:teuthology.orchestra.run.smithi112.stdout:3281 writesame 0xe29a362 thru 0xe2a8c5e data_size 0x5(0xe8fd bytes) 2024-09-04T21:54:43.078 INFO:teuthology.orchestra.run.smithi112.stdout:3282 writesame 0xcd4bffe thru 0xcd4ccfe data_size 0x1(0xd01 bytes) 2024-09-04T21:54:43.084 INFO:teuthology.orchestra.run.smithi112.stdout:3283 write 0xcaa7be4 thru 0xcab16bf (0x9adc bytes) 2024-09-04T21:54:43.093 INFO:teuthology.orchestra.run.smithi112.stdout:3284 punch from 0xa748936 to 0xa754f55, (0xc61f bytes) 2024-09-04T21:54:43.093 INFO:teuthology.orchestra.run.smithi112.stdout:3285 write 0x79c469d thru 0x79d42eb (0xfc4f bytes) 2024-09-04T21:54:43.095 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:43.095 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:43.103 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.103 INFO:teuthology.orchestra.run.smithi112.stdout:3287 write 0x1d5d6b3 thru 0x1d6cb25 (0xf473 bytes) 2024-09-04T21:54:43.112 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:43.113 INFO:teuthology.orchestra.run.smithi112.stdout:3289 read 0xd647086 thru 0xd64b86a (0x47e5 bytes) 2024-09-04T21:54:43.114 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.114 INFO:teuthology.orchestra.run.smithi112.stdout:3291 read 0x75f2095 thru 0x75f3fd2 (0x1f3e bytes) 2024-09-04T21:54:43.114 INFO:teuthology.orchestra.run.smithi112.stdout:3292 write 0x75af34a thru 0x75ba13a (0xadf1 bytes) 2024-09-04T21:54:43.124 INFO:teuthology.orchestra.run.smithi112.stdout:3296 read 0x56232a1 thru 0x562a36d (0x70cd bytes) 2024-09-04T21:54:43.124 INFO:teuthology.orchestra.run.smithi112.stdout:3297 writesame 0xae6cc30 thru 0xae75b47 data_size 0x4c(0x8f18 bytes) 2024-09-04T21:54:43.155 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:43.153+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:43.161 INFO:teuthology.orchestra.run.smithi112.stdout:3298 trunc from 0xe431f8f to 0x3718e0 2024-09-04T21:54:43.187 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.187 INFO:teuthology.orchestra.run.smithi112.stdout:3300 read 0x309d39 thru 0x3132c4 (0x958c bytes) 2024-09-04T21:54:43.188 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.218 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:43.217+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:43.226 INFO:teuthology.orchestra.run.smithi112.stdout:3304 write 0xdc8b1f1 thru 0xdc99ffc (0xee0c bytes) 2024-09-04T21:54:43.238 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.239 INFO:teuthology.orchestra.run.smithi112.stdout:3307 trunc from 0xdc99ffd to 0xe56c3d8 2024-09-04T21:54:43.250 INFO:teuthology.orchestra.run.smithi112.stdout:3308 writesame 0x7e1e637 thru 0x7e29431 data_size 0x1(0xadfb bytes) 2024-09-04T21:54:43.288 INFO:teuthology.orchestra.run.smithi112.stdout:3310 read 0x1c5abcf thru 0x1c6200c (0x743e bytes) 2024-09-04T21:54:43.288 INFO:teuthology.orchestra.run.smithi112.stdout:3312 write 0x3c43b2d thru 0x3c51788 (0xdc5c bytes) 2024-09-04T21:54:43.288 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:43.288 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:43.304 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:43.305 INFO:teuthology.orchestra.run.smithi112.stdout:3313 read 0x9b27fca thru 0x9b29b48 (0x1b7f bytes) 2024-09-04T21:54:43.305 INFO:teuthology.orchestra.run.smithi112.stdout:3314 write 0x851311b thru 0x851d60e (0xa4f4 bytes) 2024-09-04T21:54:43.317 INFO:teuthology.orchestra.run.smithi112.stdout:3315 read 0x4291300 thru 0x429a30f (0x9010 bytes) 2024-09-04T21:54:43.318 INFO:teuthology.orchestra.run.smithi112.stdout:3316 punch from 0x1c53d0b to 0x1c554a0, (0x1795 bytes) 2024-09-04T21:54:43.318 INFO:teuthology.orchestra.run.smithi112.stdout:3319 write 0xa674c43 thru 0xa6848bd (0xfc7b bytes) 2024-09-04T21:54:43.347 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:43.345+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:43.348 INFO:teuthology.orchestra.run.smithi112.stdout:3320 writesame 0xea2c1ea thru 0xea38d96 data_size 0x1(0xcbad bytes) 2024-09-04T21:54:43.376 INFO:teuthology.orchestra.run.smithi112.stdout:3321 trunc from 0xea38d97 to 0x3945554 2024-09-04T21:54:43.402 INFO:teuthology.orchestra.run.smithi112.stdout:3323 read 0x1de5b8 thru 0x1eb1fc (0xcc45 bytes) 2024-09-04T21:54:43.408 INFO:teuthology.orchestra.run.smithi112.stdout:3324 trunc from 0x3945554 to 0x1efd99f 2024-09-04T21:54:43.419 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:43.417+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:43.456 INFO:teuthology.orchestra.run.smithi112.stdout:3325 write 0xb603e6f thru 0xb60f7a7 (0xb939 bytes) 2024-09-04T21:54:43.468 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.468 INFO:teuthology.orchestra.run.smithi112.stdout:3327 punch from 0x5f1760d to 0x5f1af34, (0x3927 bytes) 2024-09-04T21:54:43.468 INFO:teuthology.orchestra.run.smithi112.stdout:3328 read 0x6b10bf8 thru 0x6b18be8 (0x7ff1 bytes) 2024-09-04T21:54:43.468 INFO:teuthology.orchestra.run.smithi112.stdout:3330 read 0x8677e4 thru 0x867a80 (0x29d bytes) 2024-09-04T21:54:43.468 INFO:teuthology.orchestra.run.smithi112.stdout:3331 write 0x7d53f43 thru 0x7d5b857 (0x7915 bytes) 2024-09-04T21:54:43.478 INFO:teuthology.orchestra.run.smithi112.stdout:3332 read 0x49752db thru 0x49816d0 (0xc3f6 bytes) 2024-09-04T21:54:43.478 INFO:teuthology.orchestra.run.smithi112.stdout:3333 read 0x4ba607f thru 0x4baac0c (0x4b8e bytes) 2024-09-04T21:54:43.478 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.478 INFO:teuthology.orchestra.run.smithi112.stdout:3335 write 0x83f1e4d thru 0x83f5f15 (0x40c9 bytes) 2024-09-04T21:54:43.479 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:43.479 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:43.486 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.486 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.495 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:43.504 INFO:teuthology.orchestra.run.smithi112.stdout:3339 trunc from 0xb60f7a8 to 0x45980a2 2024-09-04T21:54:43.520 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:43.517+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:43.551 INFO:teuthology.orchestra.run.smithi112.stdout:3340 write 0xa97890c thru 0xa9846a9 (0xbd9e bytes) 2024-09-04T21:54:43.565 INFO:teuthology.orchestra.run.smithi112.stdout:3341 writesame 0x8a5371d thru 0x8a602ee data_size 0x7(0xcbd2 bytes) 2024-09-04T21:54:43.583 INFO:teuthology.orchestra.run.smithi112.stdout:3342 write 0x6e6d688 thru 0x6e7b05a (0xd9d3 bytes) 2024-09-04T21:54:43.596 INFO:teuthology.orchestra.run.smithi112.stdout:3343 read 0x3ca4a55 thru 0x3cb0224 (0xb7d0 bytes) 2024-09-04T21:54:43.597 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.607 INFO:teuthology.orchestra.run.smithi112.stdout:3345 trunc from 0xa9846aa to 0xe345821 2024-09-04T21:54:43.617 INFO:teuthology.orchestra.run.smithi112.stdout:3346 write 0xb8e6215 thru 0xb8f570b (0xf4f7 bytes) 2024-09-04T21:54:43.620 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:43.617+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:43.626 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:43.626 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:43.635 INFO:teuthology.orchestra.run.smithi112.stdout:3347 write 0x699b63a thru 0x699ed98 (0x375f bytes) 2024-09-04T21:54:43.642 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:43.643 INFO:teuthology.orchestra.run.smithi112.stdout:3348 read 0x8b21c3f thru 0x8b2b938 (0x9cfa bytes) 2024-09-04T21:54:43.644 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.644 INFO:teuthology.orchestra.run.smithi112.stdout:3350 read 0xa025891 thru 0xa0326ca (0xce3a bytes) 2024-09-04T21:54:43.645 INFO:teuthology.orchestra.run.smithi112.stdout:3351 writesame 0xc2f2a6c thru 0xc300ef5 data_size 0x93(0xe48a bytes) 2024-09-04T21:54:43.661 INFO:teuthology.orchestra.run.smithi112.stdout:3354 read 0xb5f80ff thru 0xb601cfe (0x9c00 bytes) 2024-09-04T21:54:43.662 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.672 INFO:teuthology.orchestra.run.smithi112.stdout:3356 trunc from 0xe345821 to 0xa967e19 2024-09-04T21:54:43.696 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:54:43.714 INFO:teuthology.orchestra.run.smithi112.stdout:3357 trunc from 0xa967e19 to 0x449be92 2024-09-04T21:54:43.731 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:43.729+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:43.734 INFO:teuthology.orchestra.run.smithi112.stdout:3359 trunc from 0x449be92 to 0x3c667c1 2024-09-04T21:54:43.741 INFO:teuthology.orchestra.run.smithi112.stdout:3361 punch from 0x365d24 to 0x365ef6, (0x1d2 bytes) 2024-09-04T21:54:43.748 INFO:teuthology.orchestra.run.smithi112.stdout:3362 trunc from 0x3c667c1 to 0x78e732d 2024-09-04T21:54:43.755 INFO:teuthology.orchestra.run.smithi112.stdout:3363 write 0x696fc45 thru 0x69756c2 (0x5a7e bytes) 2024-09-04T21:54:43.762 INFO:teuthology.orchestra.run.smithi112.stdout:3364 punch from 0x3094f85 to 0x3097d4e, (0x2dc9 bytes) 2024-09-04T21:54:43.762 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.762 INFO:teuthology.orchestra.run.smithi112.stdout:3368 punch from 0x22ac390 to 0x22b8a05, (0xc675 bytes) 2024-09-04T21:54:43.762 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:43.762 INFO:teuthology.orchestra.run.smithi112.stdout:3371 write 0x27d010d thru 0x27d2ec3 (0x2db7 bytes) 2024-09-04T21:54:43.769 INFO:teuthology.orchestra.run.smithi112.stdout:3372 trunc from 0x78e732d to 0x95b7f6f 2024-09-04T21:54:43.774 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:43.776 INFO:teuthology.orchestra.run.smithi112.stdout:3375 read 0x3de809c thru 0x3df41bc (0xc121 bytes) 2024-09-04T21:54:43.796 INFO:teuthology.orchestra.run.smithi112.stdout:3376 write 0xce7e674 thru 0xce8df33 (0xf8c0 bytes) 2024-09-04T21:54:43.806 INFO:teuthology.orchestra.run.smithi112.stdout:3381 writesame 0x6a99981 thru 0x6aa57a9 data_size 0x1b(0xbe29 bytes) 2024-09-04T21:54:43.816 INFO:teuthology.orchestra.run.smithi112.stdout:3382 read 0x60f4b08 thru 0x60fd648 (0x8b41 bytes) 2024-09-04T21:54:43.816 INFO:teuthology.orchestra.run.smithi112.stdout:3383 writesame 0xf5fca4 thru 0xf63352 data_size 0x1(0x36af bytes) 2024-09-04T21:54:43.822 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:43.817+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:43.822 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:43.822 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:43.836 INFO:teuthology.orchestra.run.smithi112.stdout:3384 write 0xc145dc thru 0xc19ea6 (0x58cb bytes) 2024-09-04T21:54:43.838 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:43.839 INFO:teuthology.orchestra.run.smithi112.stdout:3385 writesame 0xcde8458 thru 0xcdf535a data_size 0x5(0xcf03 bytes) 2024-09-04T21:54:43.851 INFO:teuthology.orchestra.run.smithi112.stdout:3386 read 0xa56581c thru 0xa571ba0 (0xc385 bytes) 2024-09-04T21:54:43.864 INFO:teuthology.orchestra.run.smithi112.stdout:3387 write 0xd64ee48 thru 0xd655686 (0x683f bytes) 2024-09-04T21:54:43.871 INFO:teuthology.orchestra.run.smithi112.stdout:3389 write 0xa46159d thru 0xa46d82b (0xc28f bytes) 2024-09-04T21:54:43.883 INFO:teuthology.orchestra.run.smithi112.stdout:3391 write 0xc476322 thru 0xc481055 (0xad34 bytes) 2024-09-04T21:54:43.894 INFO:teuthology.orchestra.run.smithi112.stdout:3394 write 0x612482b thru 0x612df44 (0x971a bytes) 2024-09-04T21:54:43.903 INFO:teuthology.orchestra.run.smithi112.stdout:3395 punch from 0x4f1aa00 to 0x4f1c77b, (0x1d7b bytes) 2024-09-04T21:54:43.918 INFO:teuthology.orchestra.run.smithi112.stdout:3396 trunc from 0xd655687 to 0x47409f1 2024-09-04T21:54:43.923 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:43.921+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:43.936 INFO:teuthology.orchestra.run.smithi112.stdout:3399 read 0x9de73c thru 0x9e06ad (0x1f72 bytes) 2024-09-04T21:54:43.938 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:43.939 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:43.948 INFO:teuthology.orchestra.run.smithi112.stdout:3400 write 0x75c920e thru 0x75c9e64 (0xc57 bytes) 2024-09-04T21:54:43.953 INFO:teuthology.orchestra.run.smithi112.stdout:3401 read 0x6c7aea7 thru 0x6c87e7e (0xcfd8 bytes) 2024-09-04T21:54:43.953 INFO:teuthology.orchestra.run.smithi112.stdout:3402 read 0x71047b4 thru 0x7106639 (0x1e86 bytes) 2024-09-04T21:54:43.953 INFO:teuthology.orchestra.run.smithi112.stdout:3403 read 0xd70427 thru 0xd78d40 (0x891a bytes) 2024-09-04T21:54:43.953 INFO:teuthology.orchestra.run.smithi112.stdout:3404 write 0x160b thru 0x7668 (0x605e bytes) 2024-09-04T21:54:43.957 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:43.957 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:43.969 INFO:teuthology.orchestra.run.smithi112.stdout:3405 write 0xa225df4 thru 0xa22bbf3 (0x5e00 bytes) 2024-09-04T21:54:43.974 INFO:teuthology.orchestra.run.smithi112.stdout:3406 write 0x74aaa89 thru 0x74af05e (0x45d6 bytes) 2024-09-04T21:54:43.976 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:43.976 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:43.980 INFO:teuthology.orchestra.run.smithi112.stdout:3407 writesame 0x45c6586 thru 0x45c66bd data_size 0xc(0x138 bytes) 2024-09-04T21:54:43.984 INFO:teuthology.orchestra.run.smithi112.stdout:3408 write 0x35da60d thru 0x35e410a (0x9afe bytes) 2024-09-04T21:54:43.992 INFO:teuthology.orchestra.run.smithi112.stdout:3409 write 0x1ec9d76 thru 0x1ed7e94 (0xe11f bytes) 2024-09-04T21:54:43.996 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:43.996 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:44.001 INFO:teuthology.orchestra.run.smithi112.stdout:3410 writesame 0x3c0d03a thru 0x3c1a09b data_size 0x22(0xd062 bytes) 2024-09-04T21:54:44.009 INFO:teuthology.orchestra.run.smithi112.stdout:3411 writesame 0x8487a31 thru 0x8496592 data_size 0x53(0xeb62 bytes) 2024-09-04T21:54:44.018 INFO:teuthology.orchestra.run.smithi112.stdout:3412 punch from 0x53bd1cf to 0x53c4fba, (0x7deb bytes) 2024-09-04T21:54:44.019 INFO:teuthology.orchestra.run.smithi112.stdout:3413 writesame 0x8bd863e thru 0x8bdda02 data_size 0x1(0x53c5 bytes) 2024-09-04T21:54:44.019 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:44.019 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:44.023 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:44.021+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:44.025 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:44.025 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:44.030 INFO:teuthology.orchestra.run.smithi112.stdout:3415 read 0x87dfe4d thru 0x87e2a97 (0x2c4b bytes) 2024-09-04T21:54:44.031 INFO:teuthology.orchestra.run.smithi112.stdout:3416 punch from 0x2c695d1 to 0x2c71787, (0x81b6 bytes) 2024-09-04T21:54:44.031 INFO:teuthology.orchestra.run.smithi112.stdout:3417 read 0x18dfdff thru 0x18efcc0 (0xfec2 bytes) 2024-09-04T21:54:44.031 INFO:teuthology.orchestra.run.smithi112.stdout:3418 punch from 0x7d52d7b to 0x7d62675, (0xf8fa bytes) 2024-09-04T21:54:44.031 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.031 INFO:teuthology.orchestra.run.smithi112.stdout:3420 read 0x5939f26 thru 0x59481b7 (0xe292 bytes) 2024-09-04T21:54:44.032 INFO:teuthology.orchestra.run.smithi112.stdout:3421 writesame 0x192258b thru 0x192fa89 data_size 0x1(0xd4ff bytes) 2024-09-04T21:54:44.040 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:44.040 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:44.041 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:44.059 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:44.059 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:44.059 INFO:teuthology.orchestra.run.smithi112.stdout:3423 writesame 0xa070778 thru 0xa072538 data_size 0x1(0x1dc1 bytes) 2024-09-04T21:54:44.065 INFO:teuthology.orchestra.run.smithi112.stdout:3425 read 0x2cb06ea thru 0x2cb44ca (0x3de1 bytes) 2024-09-04T21:54:44.075 INFO:teuthology.orchestra.run.smithi112.stdout:3428 trunc from 0xa22bbf4 to 0x420c1e4 2024-09-04T21:54:44.078 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:44.079 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:44.094 INFO:teuthology.orchestra.run.smithi112.stdout:3429 punch from 0x2e725b3 to 0x2e76303, (0x3d50 bytes) 2024-09-04T21:54:44.103 INFO:teuthology.orchestra.run.smithi112.stdout:3430 trunc from 0x420c1e4 to 0x9e502ef 2024-09-04T21:54:44.111 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.111 INFO:teuthology.orchestra.run.smithi112.stdout:3432 punch from 0x132ee5a to 0x13338e3, (0x4a89 bytes) 2024-09-04T21:54:44.112 INFO:teuthology.orchestra.run.smithi112.stdout:3433 punch from 0x52f0194 to 0x52f083c, (0x6a8 bytes) 2024-09-04T21:54:44.112 INFO:teuthology.orchestra.run.smithi112.stdout:3434 punch from 0x9a57e6c to 0x9a5bede, (0x4072 bytes) 2024-09-04T21:54:44.112 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.112 INFO:teuthology.orchestra.run.smithi112.stdout:3437 punch from 0x11d34dd to 0x11dd648, (0xa16b bytes) 2024-09-04T21:54:44.124 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:44.121+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:44.126 INFO:teuthology.orchestra.run.smithi112.stdout:3438 write 0xe0d35e0 thru 0xe0db13e (0x7b5f bytes) 2024-09-04T21:54:44.133 INFO:teuthology.orchestra.run.smithi112.stdout:3439 punch from 0x24f302d to 0x24f9924, (0x68f7 bytes) 2024-09-04T21:54:44.133 INFO:teuthology.orchestra.run.smithi112.stdout:3442 write 0x91e0f6a thru 0x91eabd1 (0x9c68 bytes) 2024-09-04T21:54:44.139 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.139 INFO:teuthology.orchestra.run.smithi112.stdout:3444 punch from 0x1eb48e1 to 0x1ec008a, (0xb7a9 bytes) 2024-09-04T21:54:44.139 INFO:teuthology.orchestra.run.smithi112.stdout:3446 read 0x4bd9e86 thru 0x4be787e (0xd9f9 bytes) 2024-09-04T21:54:44.140 INFO:teuthology.orchestra.run.smithi112.stdout:3447 trunc from 0xe0db13f to 0xedcdd90 2024-09-04T21:54:44.149 INFO:teuthology.orchestra.run.smithi112.stdout:3448 write 0x7f3ef7a thru 0x7f4e16f (0xf1f6 bytes) 2024-09-04T21:54:44.161 INFO:teuthology.orchestra.run.smithi112.stdout:3449 read 0xd2d0c44 thru 0xd2d549b (0x4858 bytes) 2024-09-04T21:54:44.161 INFO:teuthology.orchestra.run.smithi112.stdout:3450 write 0x92cdb31 thru 0x92d2d2a (0x51fa bytes) 2024-09-04T21:54:44.167 INFO:teuthology.orchestra.run.smithi112.stdout:3451 writesame 0x864d51e thru 0x8656196 data_size 0x1(0x8c79 bytes) 2024-09-04T21:54:44.181 INFO:teuthology.orchestra.run.smithi112.stdout:3452 writesame 0xfd5938 thru 0xfd7ab5 data_size 0x3(0x217e bytes) 2024-09-04T21:54:44.184 INFO:teuthology.orchestra.run.smithi112.stdout:3453 read 0x72607cd thru 0x726a739 (0x9f6d bytes) 2024-09-04T21:54:44.184 INFO:teuthology.orchestra.run.smithi112.stdout:3454 write 0xd64489a thru 0xd64c2c1 (0x7a28 bytes) 2024-09-04T21:54:44.191 INFO:teuthology.orchestra.run.smithi112.stdout:3455 write 0xb082e06 thru 0xb086b79 (0x3d74 bytes) 2024-09-04T21:54:44.197 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.197 INFO:teuthology.orchestra.run.smithi112.stdout:3458 read 0x466f521 thru 0x467e7bc (0xf29c bytes) 2024-09-04T21:54:44.197 INFO:teuthology.orchestra.run.smithi112.stdout:3459 write 0xcf80e24 thru 0xcf8c627 (0xb804 bytes) 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43732 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 139264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.972659+0000", 2024-09-04T21:54:44.220 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2439893590000004, 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013349662999999999, 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43732 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.972659+0000", 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.972659+0000", 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.972660+0000", 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3399999999999997e-07 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.972665+0000", 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9779999999999996e-06 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.972695+0000", 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0244e-05 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.972707+0000", 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1399999999999999e-05 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.979324+0000", 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066171069999999997 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.980060+0000", 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073657900000000001 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.986009+0000", 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0059484209999999997 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.225 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43750 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.973250+0000", 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2433987130000004, 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012333512, 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43750 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973250+0000", 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973250+0000", 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973251+0000", 2024-09-04T21:54:44.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973256+0000", 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1370000000000001e-06 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973273+0000", 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7635e-05 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973288+0000", 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5054e-05 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.985157+0000", 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.01186824 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.985372+0000", 2024-09-04T21:54:44.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021499900000000001 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.985583+0000", 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021156699999999999 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43751 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 507904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.230 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.973303+0000", 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2433452029999996, 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012692594, 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43751 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973303+0000", 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973303+0000", 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973304+0000", 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2600000000000001e-07 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973309+0000", 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6070000000000001e-06 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973326+0000", 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7387000000000001e-05 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.973342+0000", 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5747e-05 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.985644+0000", 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012302618 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.985834+0000", 2024-09-04T21:54:44.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189798 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.985996+0000", 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016161100000000001 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44001 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1277952~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.988667+0000", 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2279811829999998, 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.014181378999999999, 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44001 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.988667+0000", 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.988667+0000", 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.988668+0000", 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1399999999999996e-07 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.988671+0000", 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4419999999999998e-06 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.988681+0000", 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3959999999999995e-06 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.988688+0000", 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2520000000000002e-06 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997588+0000", 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088995870000000005 2024-09-04T21:54:44.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997675+0000", 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7687000000000003e-05 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.002849+0000", 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0051733009999999999 2024-09-04T21:54:44.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44175 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.995613+0000", 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2210352090000001, 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012119939999999999, 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.241 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44175 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995613+0000", 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995613+0000", 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995614+0000", 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-07 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995617+0000", 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.977e-06 2024-09-04T21:54:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995626+0000", 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4669999999999993e-06 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995634+0000", 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5089999999999996e-06 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.007510+0000", 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011875783000000001 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.007591+0000", 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1321000000000003e-05 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.007733+0000", 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000142263 2024-09-04T21:54:44.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44176 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1048576~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.995879+0000", 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.220769561, 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012112608, 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.246 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44176 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995879+0000", 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995879+0000", 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995880+0000", 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3230000000000001e-06 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995885+0000", 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2399999999999998e-06 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995899+0000", 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4028e-05 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995913+0000", 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3175e-05 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.007764+0000", 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011851484000000001 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.007845+0000", 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0937000000000001e-05 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.007991+0000", 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014642100000000001 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44178 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1089536~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.995931+0000", 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2207171959999998, 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012350221, 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.251 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44178 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995931+0000", 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995931+0000", 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995932+0000", 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2600000000000002e-07 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995940+0000", 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9619999999999993e-06 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995959+0000", 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9443e-05 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995970+0000", 2024-09-04T21:54:44.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008023+0000", 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012053202000000001 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008105+0000", 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2285000000000002e-05 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008281+0000", 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017606499999999999 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44180 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1130496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.995982+0000", 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2206664549999999, 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.014844181, 2024-09-04T21:54:44.256 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44180 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995982+0000", 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995982+0000", 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995983+0000", 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7999999999999995e-07 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.995990+0000", 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2880000000000004e-06 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996002+0000", 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2148e-05 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996010+0000", 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7530000000000001e-06 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008314+0000", 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012303913 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008396+0000", 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2497999999999995e-05 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010826+0000", 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024299009999999999 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44182 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1171456~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996060+0000", 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2205884940000002, 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012720333, 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.261 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44182 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996060+0000", 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996060+0000", 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996061+0000", 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0130000000000001e-06 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996065+0000", 2024-09-04T21:54:44.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.985e-06 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996078+0000", 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2876e-05 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996086+0000", 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1969999999999996e-06 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008563+0000", 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.01247711 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008641+0000", 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7494e-05 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008780+0000", 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000139658 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44185 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1867776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996334+0000", 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2203139470000002, 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012676411, 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.266 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44185 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996334+0000", 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996334+0000", 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996336+0000", 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.11e-06 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996340+0000", 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5870000000000003e-06 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996351+0000", 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996359+0000", 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8029999999999997e-06 2024-09-04T21:54:44.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008808+0000", 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012449052 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.008887+0000", 2024-09-04T21:54:44.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9826000000000004e-05 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009011+0000", 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012344600000000001 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44186 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1908736~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996490+0000", 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2201582699999998, 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012732996999999999, 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.272 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44186 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996490+0000", 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996490+0000", 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996491+0000", 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2309999999999999e-06 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996499+0000", 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5190000000000003e-06 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996516+0000", 2024-09-04T21:54:44.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7591000000000001e-05 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996538+0000", 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.126e-05 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009038+0000", 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012500442 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009112+0000", 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4313999999999994e-05 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009223+0000", 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011064 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44188 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1949696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996676+0000", 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2199727969999996, 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012798501, 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44188 2024-09-04T21:54:44.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996676+0000", 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996676+0000", 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996677+0000", 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1090000000000001e-06 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996681+0000", 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.634e-06 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996697+0000", 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5760000000000002e-05 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996707+0000", 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5340000000000005e-06 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009248+0000", 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012541637 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009365+0000", 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011658300000000001 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009474+0000", 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109244 2024-09-04T21:54:44.281 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44190 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1212416~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996730+0000", 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2199185039999998, 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012957441, 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44190 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996730+0000", 2024-09-04T21:54:44.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996730+0000", 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996731+0000", 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1399999999999996e-07 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996734+0000", 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.405e-06 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996761+0000", 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6451000000000001e-05 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996781+0000", 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0820000000000001e-05 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009500+0000", 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012718462 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009579+0000", 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8806000000000004e-05 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009687+0000", 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108683 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44192 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1253376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996805+0000", 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2198431020000005, 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013104835, 2024-09-04T21:54:44.287 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44192 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996805+0000", 2024-09-04T21:54:44.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996805+0000", 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996806+0000", 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8800000000000002e-07 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996818+0000", 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2408999999999999e-05 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996829+0000", 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0964999999999999e-05 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996836+0000", 2024-09-04T21:54:44.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6019999999999996e-06 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009714+0000", 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.012878245999999999 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009801+0000", 2024-09-04T21:54:44.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6880999999999997e-05 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009910+0000", 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010904399999999999 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.292 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44194 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1294336~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996847+0000", 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2198018409999998, 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013276657000000001, 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.293 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44194 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996847+0000", 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996847+0000", 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996847+0000", 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996850+0000", 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7199999999999998e-06 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996857+0000", 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0779999999999998e-06 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996863+0000", 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7520000000000002e-06 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.009935+0000", 2024-09-04T21:54:44.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013072172999999999 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010017+0000", 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1928999999999994e-05 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010123+0000", 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106403 2024-09-04T21:54:44.297 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44196 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 1990656~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.996874+0000", 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2197740919999998, 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013497469999999999, 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.298 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44196 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996874+0000", 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996874+0000", 2024-09-04T21:54:44.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996875+0000", 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996877+0000", 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996885+0000", 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2269999999999996e-06 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.996892+0000", 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.99e-06 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010149+0000", 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013257102 2024-09-04T21:54:44.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010230+0000", 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1079000000000001e-05 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010372+0000", 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014200500000000001 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44198 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 2031616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.997071+0000", 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2195770279999998, 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013500857999999999, 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.303 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44198 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997071+0000", 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997071+0000", 2024-09-04T21:54:44.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997073+0000", 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3209999999999999e-06 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997077+0000", 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.25e-06 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997090+0000", 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2695e-05 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997103+0000", 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3091999999999999e-05 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010399+0000", 2024-09-04T21:54:44.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013296385000000001 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010480+0000", 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1040999999999999e-05 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010572+0000", 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2074e-05 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.307 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:44200 3.10 3:094dc410:::rbd_data.1188359b1b3c.0000000000000005:10 [read 2072576~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.997137+0000", 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.2195115479999998, 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013682045, 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.308 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44200 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997137+0000", 2024-09-04T21:54:44.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997137+0000", 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997138+0000", 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5499999999999997e-07 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997141+0000", 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7349999999999998e-06 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997153+0000", 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1525000000000001e-05 2024-09-04T21:54:44.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.997160+0000", 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3180000000000001e-06 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010598+0000", 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013437968 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010675+0000", 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7344999999999999e-05 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010819+0000", 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000143399 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.313 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:45328 3.10 3:0f20412f:::rbd_data.1188c43931da.000000000000000e:13 [read 253952~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:40.433932+0000", 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.782716631, 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011989049999999999, 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 45328 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.433932+0000", 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.433932+0000", 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.433932+0000", 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7099999999999997e-07 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.433934+0000", 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9209999999999999e-06 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.433942+0000", 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5619999999999998e-06 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.434317+0000", 2024-09-04T21:54:44.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037550499999999999 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.445630+0000", 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011312905999999999 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.445703+0000", 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3150000000000003e-05 2024-09-04T21:54:44.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.445921+0000", 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021763500000000001 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.318 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.8561526109999997, 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:54:44.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:54:44.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.325 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:44.327 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:44.328 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:44.325+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:44.328 INFO:teuthology.orchestra.run.smithi112.stdout:3460 trunc from 0xedcdd90 to 0x328432e 2024-09-04T21:54:44.328 INFO:teuthology.orchestra.run.smithi112.stdout:3461 punch from 0x2f63657 to 0x2f6ad14, (0x76bd bytes) 2024-09-04T21:54:44.328 INFO:teuthology.orchestra.run.smithi112.stdout:3462 writesame 0x5240b83 thru 0x524cca0 data_size 0x13(0xc11e bytes) 2024-09-04T21:54:44.328 INFO:teuthology.orchestra.run.smithi112.stdout:3465 trunc from 0x524cca1 to 0x8e33072 2024-09-04T21:54:44.328 INFO:teuthology.orchestra.run.smithi112.stdout:3468 punch from 0x8e03de9 to 0x8e06e45, (0x305c bytes) 2024-09-04T21:54:44.329 INFO:teuthology.orchestra.run.smithi112.stdout:3469 read 0x672e2ec thru 0x6730f9a (0x2caf bytes) 2024-09-04T21:54:44.329 INFO:teuthology.orchestra.run.smithi112.stdout:3470 writesame 0x6e7e372 thru 0x6e8a1cc data_size 0x1(0xbe5b bytes) 2024-09-04T21:54:44.329 INFO:teuthology.orchestra.run.smithi112.stdout:3471 writesame 0x94e8004 thru 0x94f6bbe data_size 0x25(0xebbb bytes) 2024-09-04T21:54:44.329 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.329 INFO:teuthology.orchestra.run.smithi112.stdout:3473 write 0x572f94 thru 0x573cdf (0xd4c bytes) 2024-09-04T21:54:44.343 INFO:teuthology.orchestra.run.smithi112.stdout:3474 writesame 0x9b17b3c thru 0x9b206b2 data_size 0x3(0x8b77 bytes) 2024-09-04T21:54:44.346 INFO:teuthology.orchestra.run.smithi112.stdout:3477 trunc from 0x9b206b3 to 0xad4bd61 2024-09-04T21:54:44.361 INFO:teuthology.orchestra.run.smithi112.stdout:3478 punch from 0x95e2b06 to 0x95ec31c, (0x9816 bytes) 2024-09-04T21:54:44.362 INFO:teuthology.orchestra.run.smithi112.stdout:3479 read 0x95980f6 thru 0x959b1b7 (0x30c2 bytes) 2024-09-04T21:54:44.362 INFO:teuthology.orchestra.run.smithi112.stdout:3481 trunc from 0xad4bd61 to 0xaad9c48 2024-09-04T21:54:44.395 INFO:teuthology.orchestra.run.smithi112.stdout:3482 write 0xcf20720 thru 0xcf2e8a4 (0xe185 bytes) 2024-09-04T21:54:44.411 INFO:teuthology.orchestra.run.smithi112.stdout:3483 read 0x383fcb1 thru 0x38402d8 (0x628 bytes) 2024-09-04T21:54:44.412 INFO:teuthology.orchestra.run.smithi112.stdout:3484 punch from 0x87d5b46 to 0x87d7735, (0x1bef bytes) 2024-09-04T21:54:44.412 INFO:teuthology.orchestra.run.smithi112.stdout:3486 punch from 0x392f6e6 to 0x393bef0, (0xc80a bytes) 2024-09-04T21:54:44.412 INFO:teuthology.orchestra.run.smithi112.stdout:3487 punch from 0x40829c6 to 0x4085cae, (0x32e8 bytes) 2024-09-04T21:54:44.412 INFO:teuthology.orchestra.run.smithi112.stdout:3488 writesame 0x3486fde thru 0x348ec22 data_size 0x1(0x7c45 bytes) 2024-09-04T21:54:44.427 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:44.425+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:44.437 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.437 INFO:teuthology.orchestra.run.smithi112.stdout:3491 read 0xbe431fd thru 0xbe4fb15 (0xc919 bytes) 2024-09-04T21:54:44.437 INFO:teuthology.orchestra.run.smithi112.stdout:3492 write 0xb51c931 thru 0xb52bdbb (0xf48b bytes) 2024-09-04T21:54:44.453 INFO:teuthology.orchestra.run.smithi112.stdout:3493 punch from 0x651a485 to 0x65241bf, (0x9d3a bytes) 2024-09-04T21:54:44.453 INFO:teuthology.orchestra.run.smithi112.stdout:3495 punch from 0xbeef157 to 0xbef5a4c, (0x68f5 bytes) 2024-09-04T21:54:44.453 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.453 INFO:teuthology.orchestra.run.smithi112.stdout:3497 read 0x56cc586 thru 0x56d642a (0x9ea5 bytes) 2024-09-04T21:54:44.453 INFO:teuthology.orchestra.run.smithi112.stdout:3498 read 0x69b2a29 thru 0x69b57ae (0x2d86 bytes) 2024-09-04T21:54:44.453 INFO:teuthology.orchestra.run.smithi112.stdout:3499 writesame 0x9a3bf98 thru 0x9a450b8 data_size 0x1(0x9121 bytes) 2024-09-04T21:54:44.483 INFO:teuthology.orchestra.run.smithi112.stdout:3502 write 0x36d6539 thru 0x36de006 (0x7ace bytes) 2024-09-04T21:54:44.490 INFO:teuthology.orchestra.run.smithi112.stdout:3503 punch from 0x42f6e83 to 0x42fec51, (0x7dce bytes) 2024-09-04T21:54:44.490 INFO:teuthology.orchestra.run.smithi112.stdout:3505 writesame 0xc0a208d thru 0xc0ab5be data_size 0x2(0x9532 bytes) 2024-09-04T21:54:44.510 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:44.510 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:44.510 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:54:44.510 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:44.525 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:54:44.546 INFO:teuthology.orchestra.run.smithi112.stdout:3506 trunc from 0xcf2e8a5 to 0x286d552 2024-09-04T21:54:44.567 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:44.565+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:44.616 INFO:teuthology.orchestra.run.smithi112.stdout:3507 write 0xe186be6 thru 0xe19604a (0xf465 bytes) 2024-09-04T21:54:44.629 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:44.625+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:44.632 INFO:teuthology.orchestra.run.smithi112.stdout:3508 writesame 0x758df0d thru 0x7595311 data_size 0x1(0x7405 bytes) 2024-09-04T21:54:44.657 INFO:teuthology.orchestra.run.smithi112.stdout:3509 write 0x2277284 thru 0x2284723 (0xd4a0 bytes) 2024-09-04T21:54:44.670 INFO:teuthology.orchestra.run.smithi112.stdout:3510 read 0x2cc3559 thru 0x2cd15ed (0xe095 bytes) 2024-09-04T21:54:44.670 INFO:teuthology.orchestra.run.smithi112.stdout:3512 write 0xdc6a435 thru 0xdc6bc13 (0x17df bytes) 2024-09-04T21:54:44.673 INFO:teuthology.orchestra.run.smithi112.stdout:3513 read 0xbb8d02a thru 0xbb8d3d9 (0x3b0 bytes) 2024-09-04T21:54:44.673 INFO:teuthology.orchestra.run.smithi112.stdout:3516 read 0x545f1d4 thru 0x546617d (0x6faa bytes) 2024-09-04T21:54:44.673 INFO:teuthology.orchestra.run.smithi112.stdout:3517 write 0xb4e31ba thru 0xb4f007f (0xcec6 bytes) 2024-09-04T21:54:44.694 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:44.695 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:44.695 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:44.695 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:44.695 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:44.709 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:54:44.718 INFO:teuthology.orchestra.run.smithi112.stdout:3519 trunc from 0xe19604b to 0x1b5bebc 2024-09-04T21:54:44.730 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:44.729+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:44.752 INFO:teuthology.orchestra.run.smithi112.stdout:3520 trunc from 0x1b5bebc to 0x438a96 2024-09-04T21:54:44.777 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.777 INFO:teuthology.orchestra.run.smithi112.stdout:3522 punch from 0x3783d to 0x41a96, (0xa259 bytes) 2024-09-04T21:54:44.777 INFO:teuthology.orchestra.run.smithi112.stdout:3523 read 0x348090 thru 0x34dd4a (0x5cbb bytes) 2024-09-04T21:54:44.778 INFO:teuthology.orchestra.run.smithi112.stdout:3524 read 0x101921 thru 0x10b9ed (0xa0cd bytes) 2024-09-04T21:54:44.798 INFO:teuthology.orchestra.run.smithi112.stdout:3525 trunc from 0x438a96 to 0x7b31c33 2024-09-04T21:54:44.826 INFO:teuthology.orchestra.run.smithi112.stdout:3526 write 0x9a86fa0 thru 0x9a96e75 (0xfed6 bytes) 2024-09-04T21:54:44.830 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:44.829+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:3529 read 0x7c78c3c thru 0x7c872ea (0xe6af bytes) 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:3531 read 0x62750ea thru 0x6282977 (0xd88e bytes) 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:3538 read 0x3900c5a thru 0x390831d (0x76c4 bytes) 2024-09-04T21:54:44.842 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.863 INFO:teuthology.orchestra.run.smithi112.stdout:3544 write 0xc9e72af thru 0xc9f62e5 (0xf037 bytes) 2024-09-04T21:54:44.880 INFO:teuthology.orchestra.run.smithi112.stdout:3545 writesame 0x1577697 thru 0x1581f80 data_size 0x3(0xa8ea bytes) 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.264130078000001, 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:54:44.881 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:54:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:44.886 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.253124834999999, 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:54:44.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.891 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7094420610000007, 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:54:44.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.896 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7094180849999994, 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:54:44.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7093773989999992, 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:54:44.901 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:54:44.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7093439999999998, 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:54:44.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:54:44.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:54:44.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7093007290000006, 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:54:44.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.914 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7092560480000003, 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.918 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7091948919999993, 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.919 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:54:44.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:54:44.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7091648510000006, 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.923 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7091200799999999, 2024-09-04T21:54:44.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:54:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:54:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:54:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:54:44.934 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7090717879999993, 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.935 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:44.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:54:44.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:54:44.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:44.940 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7090361430000005, 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:54:44.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7087869270000002, 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:54:44.945 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:54:44.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7087480579999994, 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.950 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:54:44.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.954 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7036013130000001, 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.955 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:44.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:54:44.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7035580540000002, 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.960 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:54:44.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.964 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7035141829999993, 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:54:44.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:54:44.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7034779049999997, 2024-09-04T21:54:44.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:44.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:54:44.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:54:44.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:44.974 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7034517989999998, 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:44.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:54:44.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:54:44.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:54:44.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:44.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:44.979 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:44.981 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:44.977+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:44.981 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:44.982 INFO:teuthology.orchestra.run.smithi112.stdout:3546 read 0x9875081 thru 0x987d273 (0x81f3 bytes) 2024-09-04T21:54:44.982 INFO:teuthology.orchestra.run.smithi112.stdout:3547 punch from 0x4a9595c to 0x4a9883c, (0x2ee0 bytes) 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3549 write 0x12e048f thru 0x12ec7a0 (0xc312 bytes) 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3550 read 0x3c7a686 thru 0x3c83906 (0x9281 bytes) 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3551 read 0x403d028 thru 0x40457c0 (0x8799 bytes) 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3553 write 0x6f62368 thru 0x6f69690 (0x7329 bytes) 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3556 trunc from 0xc9f62e6 to 0xa4cea98 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3557 writesame 0x58ae107 thru 0x58ae9ef data_size 0x1(0x8e9 bytes) 2024-09-04T21:54:44.983 INFO:teuthology.orchestra.run.smithi112.stdout:3558 write 0x3b2f8f4 thru 0x3b3dc6b (0xe378 bytes) 2024-09-04T21:54:45.007 INFO:teuthology.orchestra.run.smithi112.stdout:3559 write 0xa7bf8a2 thru 0xa7c730d (0x7a6c bytes) 2024-09-04T21:54:45.026 INFO:teuthology.orchestra.run.smithi112.stdout:3560 trunc from 0xa7c730e to 0x59bd0a5 2024-09-04T21:54:45.056 INFO:teuthology.orchestra.run.smithi112.stdout:3563 punch from 0x5b4bc9 to 0x5c0a85, (0xbebc bytes) 2024-09-04T21:54:45.056 INFO:teuthology.orchestra.run.smithi112.stdout:3564 punch from 0x1dec830 to 0x1df6615, (0x9de5 bytes) 2024-09-04T21:54:45.056 INFO:teuthology.orchestra.run.smithi112.stdout:3565 punch from 0x36ed2a9 to 0x36f11da, (0x3f31 bytes) 2024-09-04T21:54:45.056 INFO:teuthology.orchestra.run.smithi112.stdout:3567 read 0x2278858 thru 0x22815e1 (0x8d8a bytes) 2024-09-04T21:54:45.057 INFO:teuthology.orchestra.run.smithi112.stdout:3568 read 0x518fe2 thru 0x5256e4 (0xc703 bytes) 2024-09-04T21:54:45.057 INFO:teuthology.orchestra.run.smithi112.stdout:3571 write 0x294f0e9 thru 0x295c223 (0xd13b bytes) 2024-09-04T21:54:45.081 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:45.081+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:45.101 INFO:teuthology.orchestra.run.smithi112.stdout:3572 writesame 0xecfdd21 thru 0xed044f4 data_size 0x1e(0x67d4 bytes) 2024-09-04T21:54:45.101 INFO:teuthology.orchestra.run.smithi112.stdout:3573 write 0xd359fbe thru 0xd35b43f (0x1482 bytes) 2024-09-04T21:54:45.107 INFO:teuthology.orchestra.run.smithi112.stdout:3574 write 0xa28b9a9 thru 0xa29b8b8 (0xff10 bytes) 2024-09-04T21:54:45.121 INFO:teuthology.orchestra.run.smithi112.stdout:3575 write 0x479e696 thru 0x47a7489 (0x8df4 bytes) 2024-09-04T21:54:45.131 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:45.131 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:45.133 INFO:teuthology.orchestra.run.smithi112.stdout:3576 read 0x3468c59 thru 0x346db8f (0x4f37 bytes) 2024-09-04T21:54:45.133 INFO:teuthology.orchestra.run.smithi112.stdout:3578 write 0x9c8cdea thru 0x9c9816b (0xb382 bytes) 2024-09-04T21:54:45.146 INFO:teuthology.orchestra.run.smithi112.stdout:3580 punch from 0xd9ec484 to 0xd9eefcd, (0x2b49 bytes) 2024-09-04T21:54:45.146 INFO:teuthology.orchestra.run.smithi112.stdout:3581 write 0xdbf9712 thru 0xdbff09c (0x598b bytes) 2024-09-04T21:54:45.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:45.155 INFO:teuthology.orchestra.run.smithi112.stdout:3582 write 0x1258def thru 0x1266b8a (0xdd9c bytes) 2024-09-04T21:54:45.165 INFO:teuthology.orchestra.run.smithi112.stdout:3584 write 0xa38b06e thru 0xa3924f7 (0x748a bytes) 2024-09-04T21:54:45.173 INFO:teuthology.orchestra.run.smithi112.stdout:3585 clone 18 order 21 su 32768 sc 6 2024-09-04T21:54:45.181 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:45.181+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:45.278 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:45.278 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:45.278 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:45.278 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:45.278 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:45.282 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:45.281+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:45.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:45.383 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:45.381+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799231231, 2024-09-04T21:54:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799204445999997, 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799175513000002, 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799151866000003, 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799132084999997, 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799112690000001, 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799093417000002, 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.503 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:45.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.507 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799074195000003, 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.508 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:45.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.512 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799055357999997, 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:45.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799035955000001, 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.517 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:45.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:45.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.799014898999999, 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.522 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798982721000002, 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798956500000003, 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798928291000003, 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:45.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.540 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798908251, 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.541 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:45.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:45.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798888869000002, 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:45.548 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:45.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798865280999998, 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.553 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:45.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:45.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:45.557 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798844963999997, 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798825411999999, 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.798805653000002, 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:45.571 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:45.572 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:45.573 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone17 from 0xed044f5 (overlap 0x3718e0) to 0x5827c 2024-09-04T21:54:45.573 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:45.569+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:45.673 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:45.669+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:45.729 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:54:45.751 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:45.751 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:45.751 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:45.751 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:45.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:45.784 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-09-04T21:54:45.784 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:45.781+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:45.874 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:45.873+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:45.937 INFO:teuthology.orchestra.run.smithi112.stdout:3586 writesame 0xbc86dc8 thru 0xbc8db76 data_size 0x1(0x6daf bytes) 2024-09-04T21:54:45.939 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:45.939 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:45.957 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:45.968 INFO:teuthology.orchestra.run.smithi112.stdout:3587 punch from 0x9c6827c to 0x9c7171b, (0x949f bytes) 2024-09-04T21:54:45.968 INFO:teuthology.orchestra.run.smithi112.stdout:3590 read 0x663b0a2 thru 0x663e294 (0x31f3 bytes) 2024-09-04T21:54:45.970 INFO:teuthology.orchestra.run.smithi112.stdout:3591 trunc from 0xed044f5 to 0xd7a2bbe 2024-09-04T21:54:45.975 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:45.973+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:46.007 INFO:teuthology.orchestra.run.smithi112.stdout:3592 writesame 0x44afef1 thru 0x44b6aa9 data_size 0x17(0x6bb9 bytes) 2024-09-04T21:54:46.018 INFO:teuthology.orchestra.run.smithi112.stdout:3593 read 0x8343876 thru 0x83511fd (0xd988 bytes) 2024-09-04T21:54:46.018 INFO:teuthology.orchestra.run.smithi112.stdout:3594 read 0x6398ad0 thru 0x639f23f (0x6770 bytes) 2024-09-04T21:54:46.018 INFO:teuthology.orchestra.run.smithi112.stdout:3595 read 0x1ea08a8 thru 0x1ea1403 (0xb5c bytes) 2024-09-04T21:54:46.019 INFO:teuthology.orchestra.run.smithi112.stdout:3596 write 0x41ca4e4 thru 0x41d0cec (0x6809 bytes) 2024-09-04T21:54:46.031 INFO:teuthology.orchestra.run.smithi112.stdout:3598 read 0xa4e9fdd thru 0xa4f26d8 (0x86fc bytes) 2024-09-04T21:54:46.032 INFO:teuthology.orchestra.run.smithi112.stdout:3599 read 0xd72c919 thru 0xd736c85 (0xa36d bytes) 2024-09-04T21:54:46.037 INFO:teuthology.orchestra.run.smithi112.stdout:3600 trunc from 0xd7a2bbe to 0xaa3e000 2024-09-04T21:54:46.076 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:46.073+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:46.080 INFO:teuthology.orchestra.run.smithi112.stdout:3601 write 0x14a95d6 thru 0x14ab1f9 (0x1c24 bytes) 2024-09-04T21:54:46.094 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.094 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.094 INFO:teuthology.orchestra.run.smithi112.stdout:3606 write 0x5f72404 thru 0x5f81e63 (0xfa60 bytes) 2024-09-04T21:54:46.099 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:46.100 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 43.247648237, 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:46.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 43.121600974000003, 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:46.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:46.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:46.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.925047335999999, 2024-09-04T21:54:46.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:46.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:46.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:46.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:46.120 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.897657838000001, 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:46.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.874393367000003, 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.125 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.130 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.846635577000001, 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.131 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:46.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:46.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.846520015000003, 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.54634523, 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:46.147 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:46.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.387212677999997, 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.153 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:46.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:46.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.386986620000002, 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:46.159 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:46.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.909773551999997, 2024-09-04T21:54:46.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:46.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:46.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.680385440999999, 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:46.171 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:46.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:46.177 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.382194800000001, 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:46.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:46.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:46.183 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.382081945000003, 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:46.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:46.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:46.189 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.372420976999997, 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:46.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:46.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.352571251999997, 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.194 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:46.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.202342565999999, 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:46.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.536015661, 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.532876475000002, 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:46.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.514565484999999, 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:46.215 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:46.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:46.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:46.219 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:46.220 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:46.221 INFO:teuthology.orchestra.run.smithi112.stdout:3607 read 0x95d395b thru 0x95d8646 (0x4cec bytes) 2024-09-04T21:54:46.221 INFO:teuthology.orchestra.run.smithi112.stdout:3608 read 0x23c7b95 thru 0x23d16fe (0x9b6a bytes) 2024-09-04T21:54:46.221 INFO:teuthology.orchestra.run.smithi112.stdout:3610 read 0x878e5a thru 0x87ef02 (0x60a9 bytes) 2024-09-04T21:54:46.222 INFO:teuthology.orchestra.run.smithi112.stdout:3612 flatten 2024-09-04T21:54:46.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:46.223 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:46.221+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:46.322 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:46.317+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:46.327 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:46.327 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:54:46.355 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:46.355 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:46.364 INFO:teuthology.orchestra.run.smithi112.stdout:3613 write 0xebe633c thru 0xebf5186 (0xee4b bytes) 2024-09-04T21:54:46.370 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.370 INFO:teuthology.orchestra.run.smithi112.stdout:3616 write 0x7046666 thru 0x704a563 (0x3efe bytes) 2024-09-04T21:54:46.375 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:46.375 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:46.376 INFO:teuthology.orchestra.run.smithi112.stdout:3619 write 0x6f6276f thru 0x6f64898 (0x212a bytes) 2024-09-04T21:54:46.379 INFO:teuthology.orchestra.run.smithi112.stdout:3620 write 0x426308c thru 0x426ff9b (0xcf10 bytes) 2024-09-04T21:54:46.382 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.382 INFO:teuthology.orchestra.run.smithi112.stdout:3622 write 0x8f32469 thru 0x8f37b4c (0x56e4 bytes) 2024-09-04T21:54:46.386 INFO:teuthology.orchestra.run.smithi112.stdout:3624 read 0x54bad88 thru 0x54bdb70 (0x2de9 bytes) 2024-09-04T21:54:46.386 INFO:teuthology.orchestra.run.smithi112.stdout:3626 write 0xc9b1c3c thru 0xc9be353 (0xc718 bytes) 2024-09-04T21:54:46.390 INFO:teuthology.orchestra.run.smithi112.stdout:3627 read 0x959fa6 thru 0x963230 (0x928b bytes) 2024-09-04T21:54:46.391 INFO:teuthology.orchestra.run.smithi112.stdout:3628 write 0x6a21017 thru 0x6a27eb2 (0x6e9c bytes) 2024-09-04T21:54:46.393 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:46.394 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:46.394 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.395 INFO:teuthology.orchestra.run.smithi112.stdout:3631 writesame 0xb9f8d53 thru 0xba05df8 data_size 0x11(0xd0a6 bytes) 2024-09-04T21:54:46.406 INFO:teuthology.orchestra.run.smithi112.stdout:3634 trunc from 0xebf5187 to 0xb78395d 2024-09-04T21:54:46.414 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:46.414 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:46.416 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:46.416 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:46.418 INFO:teuthology.orchestra.run.smithi112.stdout:3635 read 0x38c3392 thru 0x38c533a (0x1fa9 bytes) 2024-09-04T21:54:46.419 INFO:teuthology.orchestra.run.smithi112.stdout:3638 writesame 0x9799e22 thru 0x97a7f2f data_size 0x1(0xe10e bytes) 2024-09-04T21:54:46.423 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:46.421+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:46.430 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:46.434 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:46.435 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:46.441 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.453 INFO:teuthology.orchestra.run.smithi112.stdout:3640 trunc from 0xb78395d to 0x297d057 2024-09-04T21:54:46.460 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:46.483 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:46.484 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:46.491 INFO:teuthology.orchestra.run.smithi112.stdout:3641 write 0x9ce2229 thru 0x9ce4aaa (0x2882 bytes) 2024-09-04T21:54:46.494 INFO:teuthology.orchestra.run.smithi112.stdout:3643 writesame 0x4c4e230 thru 0x4c5613a data_size 0x25(0x7f0b bytes) 2024-09-04T21:54:46.504 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.504 INFO:teuthology.orchestra.run.smithi112.stdout:3645 writesame 0x3345aaa thru 0x3353207 data_size 0x6(0xd75e bytes) 2024-09-04T21:54:46.512 INFO:teuthology.orchestra.run.smithi112.stdout:3646 read 0x2543f42 thru 0x254daea (0x9ba9 bytes) 2024-09-04T21:54:46.513 INFO:teuthology.orchestra.run.smithi112.stdout:3647 punch from 0x9313080 to 0x9319a90, (0x6a10 bytes) 2024-09-04T21:54:46.524 INFO:teuthology.orchestra.run.smithi112.stdout:3648 trunc from 0x9ce4aab to 0x739ab7 2024-09-04T21:54:46.524 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:46.521+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:46.549 INFO:teuthology.orchestra.run.smithi112.stdout:3651 read 0x4aae98 thru 0x4b8316 (0xd47f bytes) 2024-09-04T21:54:46.549 INFO:teuthology.orchestra.run.smithi112.stdout:3652 read 0x2f2e03 thru 0x3028c0 (0xfabe bytes) 2024-09-04T21:54:46.550 INFO:teuthology.orchestra.run.smithi112.stdout:3653 read 0x1ec648 thru 0x1f09c3 (0x437c bytes) 2024-09-04T21:54:46.551 INFO:teuthology.orchestra.run.smithi112.stdout:3654 read 0x5ce7b2 thru 0x5da789 (0xbfd8 bytes) 2024-09-04T21:54:46.552 INFO:teuthology.orchestra.run.smithi112.stdout:3656 punch from 0x25061c to 0x25e70e, (0xe0f2 bytes) 2024-09-04T21:54:46.552 INFO:teuthology.orchestra.run.smithi112.stdout:3657 read 0x62bf50 thru 0x62e173 (0x2224 bytes) 2024-09-04T21:54:46.552 INFO:teuthology.orchestra.run.smithi112.stdout:3658 read 0x8c26b thru 0x9c1bd (0xff53 bytes) 2024-09-04T21:54:46.553 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.559 INFO:teuthology.orchestra.run.smithi112.stdout:3661 trunc from 0x739ab7 to 0x4b2b96a 2024-09-04T21:54:46.563 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:46.563 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:46.568 INFO:teuthology.orchestra.run.smithi112.stdout:3662 write 0x46dbfdb thru 0x46e1841 (0x5867 bytes) 2024-09-04T21:54:46.575 INFO:teuthology.orchestra.run.smithi112.stdout:3666 write 0x3d16bd8 thru 0x3d2461d (0xda46 bytes) 2024-09-04T21:54:46.577 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:46.582 INFO:teuthology.orchestra.run.smithi112.stdout:3667 punch from 0x5ba7f4 to 0x5c3e82, (0x968e bytes) 2024-09-04T21:54:46.582 INFO:teuthology.orchestra.run.smithi112.stdout:3668 write 0x40a1d79 thru 0x40ac98c (0xac14 bytes) 2024-09-04T21:54:46.588 INFO:teuthology.orchestra.run.smithi112.stdout:3669 trunc from 0x4b2b96a to 0x807da4d 2024-09-04T21:54:46.601 INFO:teuthology.orchestra.run.smithi112.stdout:3670 trunc from 0x807da4d to 0xc3042c8 2024-09-04T21:54:46.610 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:46.610 INFO:teuthology.orchestra.run.smithi112.stdout:3673 read 0x14ff7f thru 0x150b79 (0xbfb bytes) 2024-09-04T21:54:46.610 INFO:teuthology.orchestra.run.smithi112.stdout:3675 punch from 0x3a03b99 to 0x3a0f2c4, (0xb72b bytes) 2024-09-04T21:54:46.610 INFO:teuthology.orchestra.run.smithi112.stdout:3676 write 0xa5aa55d thru 0xa5b2a77 (0x851b bytes) 2024-09-04T21:54:46.620 INFO:teuthology.orchestra.run.smithi112.stdout:3677 write 0x42394cf thru 0x4245ffa (0xcb2c bytes) 2024-09-04T21:54:46.625 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:46.621+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:46.628 INFO:teuthology.orchestra.run.smithi112.stdout:3678 write 0x36328a6 thru 0x3634605 (0x1d60 bytes) 2024-09-04T21:54:46.634 INFO:teuthology.orchestra.run.smithi112.stdout:3679 read 0x6ac1b82 thru 0x6ac63b7 (0x4836 bytes) 2024-09-04T21:54:46.635 INFO:teuthology.orchestra.run.smithi112.stdout:3680 write 0x143b64 thru 0x1503f4 (0xc891 bytes) 2024-09-04T21:54:46.646 INFO:teuthology.orchestra.run.smithi112.stdout:3681 trunc from 0xc3042c8 to 0x62400e3 2024-09-04T21:54:46.668 INFO:teuthology.orchestra.run.smithi112.stdout:3683 clone 19 order 18 su 16384 sc 12 2024-09-04T21:54:46.693 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:46.693 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:46.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:46.711 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:54:46.711 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:54:46.711 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 3, 4, 5, 6, 7] 2024-09-04T21:54:46.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-04T21:54:46.726 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:46.721+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:46.827 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:46.825+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:46.856 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:46.856 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:46.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:46.928 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:46.925+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:46.980 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:46.980 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:46.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:47.030 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:47.025+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:47.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-04T21:54:47.049+0000 7f1d9e303640 -1 mon.a@0(leader).osd e63 definitely_dead 0 2024-09-04T21:54:47.131 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:47.129+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:47.151 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:47.151 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:47.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:47.232 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:47.229+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:47.299 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:47.299 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:47.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:47.333 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:47.333+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:marked out osd.0. 2024-09-04T21:54:47.434 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:47.433+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:47.445 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:47.445 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:47.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:47.535 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:47.533+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:47.585 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:47.585 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:47.599 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:47.643 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:47.641+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:47.737 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:47.737+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:47.767 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:47.767 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:47.781 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:47.838 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:47.837+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:47.901 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:47.902 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:47.915 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:47.939 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:47.937+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:48.037 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:48.038 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:48.040 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:48.037+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:48.051 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:48.143 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:48.141+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:48.216 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:48.217 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:48.230 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:48.242 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:48.241+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:48.342 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:48.337+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:48.372 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:54:48.400 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:48.400 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:48.414 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:48.459 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:48.457+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:48.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:48.544 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:48.541+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:48.584 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:48.585 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:48.599 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:48.645 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:48.641+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:48.648 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:48.648 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:48.665 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:48.665 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:48.681 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:48.681 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:48.698 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:48.698 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:48.716 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:48.716 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:48.734 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:48.734 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:48.745 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:48.745+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:48.754 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:48.754 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:48.767 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:48.767 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:48.774 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:48.774 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:48.782 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:54:48.847 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:48.845+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:48.899 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:48.899 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:54:48.899 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:54:48.899 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:54:48.899 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:48.913 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:54:48.959 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:48.957+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:49.032 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:54:49.032 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:54:49.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:54:49.032 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:54:49.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.415558836999999, 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.033 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:54:49.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:54:49.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.404553593999999, 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:54:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:54:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860870820000001, 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860846843999999, 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:54:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:54:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:54:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860806158000001, 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:54:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:54:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:54:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:54:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860772759, 2024-09-04T21:54:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:54:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:54:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860729488, 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:54:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:54:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:54:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860684807, 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860623651000001, 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:54:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:54:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.86059361, 2024-09-04T21:54:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:54:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:54:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860548839, 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:54:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860500547000001, 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:54:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860464902, 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:54:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:54:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860215686, 2024-09-04T21:54:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.860176816999999, 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.855030072, 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:54:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.854986813, 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:54:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.854942941999999, 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:54:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.854906664, 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:54:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.854880558, 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:54:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:54:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:54:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:54:49.127 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:54:49.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:49.129 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:49.125+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:49.229 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:49.229+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:49.261 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:49.261 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:49.261 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:49.262 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:49.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:49.364 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:54:49.365 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:49.365+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:49.431 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:49.429+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:49.477 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:49.478 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:49.478 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:49.478 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:49.478 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:49.496 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:49.532 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:49.529+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:49.634 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:49.629+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:49.637 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961871035000001, 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.638 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:49.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961844249999999, 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:49.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:49.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961815317000003, 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:49.647 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:49.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961791669999997, 2024-09-04T21:54:49.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:49.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:49.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961771888999998, 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.657 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:49.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:49.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961752494000002, 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:49.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961733221000003, 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.667 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:49.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.671 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961713998999997, 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:49.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:49.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:49.676 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961695161999998, 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:49.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961675759000002, 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.681 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:49.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961654703000001, 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.686 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961622525000003, 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:49.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961596303999997, 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961568094999997, 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961548055000002, 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961528672999997, 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961505084999999, 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961484767999998, 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961465216000001, 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.961445456999996, 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:49.736 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-09-04T21:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout:3685 read 0x519bece thru 0x519d7ce (0x1901 bytes) 2024-09-04T21:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout:3686 read 0x53329cc thru 0x5334e61 (0x2496 bytes) 2024-09-04T21:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout:3687 read 0x182acf1 thru 0x1833b6b (0x8e7b bytes) 2024-09-04T21:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout:3689 write 0xcd7287 thru 0xcd9467 (0x21e1 bytes) 2024-09-04T21:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout:3690 read 0x14a7f86 thru 0x14a8b98 (0xc13 bytes) 2024-09-04T21:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout:3691 trunc from 0x62400e3 to 0x361ca 2024-09-04T21:54:49.738 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:49.733+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:49.774 INFO:teuthology.orchestra.run.smithi112.stdout:3693 write 0x1f586f8 thru 0x1f65dbf (0xd6c8 bytes) 2024-09-04T21:54:49.786 INFO:teuthology.orchestra.run.smithi112.stdout:3694 read 0x10e87ee thru 0x10edc06 (0x5419 bytes) 2024-09-04T21:54:49.786 INFO:teuthology.orchestra.run.smithi112.stdout:3695 punch from 0x35ed to 0xb45b, (0x7e6e bytes) 2024-09-04T21:54:49.786 INFO:teuthology.orchestra.run.smithi112.stdout:3696 trunc from 0x1f65dc0 to 0x27b9f43 2024-09-04T21:54:49.795 INFO:teuthology.orchestra.run.smithi112.stdout:3697 write 0x8a4ea3 thru 0x8b1346 (0xc4a4 bytes) 2024-09-04T21:54:49.816 INFO:teuthology.orchestra.run.smithi112.stdout:3699 writesame 0x32abe41 thru 0x32b4630 data_size 0x6(0x87f0 bytes) 2024-09-04T21:54:49.829 INFO:teuthology.orchestra.run.smithi112.stdout:3700 write 0x5e25f25 thru 0x5e3066d (0xa749 bytes) 2024-09-04T21:54:49.837 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:49.837+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:49.850 INFO:teuthology.orchestra.run.smithi112.stdout:3702 writesame 0x7f4e15d thru 0x7f58c72 data_size 0x2(0xab16 bytes) 2024-09-04T21:54:49.850 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:49.852 INFO:teuthology.orchestra.run.smithi112.stdout:3705 trunc from 0x7f58c73 to 0x969f824 2024-09-04T21:54:49.861 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:49.861 INFO:teuthology.orchestra.run.smithi112.stdout:3707 writesame 0x1f6de5c thru 0x1f764db data_size 0x40(0x8680 bytes) 2024-09-04T21:54:49.866 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:49.866 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:49.866 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:49.866 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:49.870 INFO:teuthology.orchestra.run.smithi112.stdout:3708 write 0x43538a1 thru 0x4355532 (0x1c92 bytes) 2024-09-04T21:54:49.875 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:49.882 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:49.887 INFO:teuthology.orchestra.run.smithi112.stdout:3710 write 0xba09352 thru 0xba10dd9 (0x7a88 bytes) 2024-09-04T21:54:49.895 INFO:teuthology.orchestra.run.smithi112.stdout:3711 write 0x9c3cf28 thru 0x9c3f9b0 (0x2a89 bytes) 2024-09-04T21:54:49.901 INFO:teuthology.orchestra.run.smithi112.stdout:3716 write 0x7778cca thru 0x7786211 (0xd548 bytes) 2024-09-04T21:54:49.909 INFO:teuthology.orchestra.run.smithi112.stdout:3717 punch from 0xa6cbb32 to 0xa6d8694, (0xcb62 bytes) 2024-09-04T21:54:49.910 INFO:teuthology.orchestra.run.smithi112.stdout:3718 read 0x7e5167 thru 0x7f4292 (0xf12c bytes) 2024-09-04T21:54:49.910 INFO:teuthology.orchestra.run.smithi112.stdout:3719 read 0x906f4e3 thru 0x907a83a (0xb358 bytes) 2024-09-04T21:54:49.918 INFO:teuthology.orchestra.run.smithi112.stdout:3720 trunc from 0xba10dda to 0x6ccdd0a 2024-09-04T21:54:49.938 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:49.933+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:49.949 INFO:teuthology.orchestra.run.smithi112.stdout:3721 write 0xbe51bf3 thru 0xbe587f9 (0x6c07 bytes) 2024-09-04T21:54:49.956 INFO:teuthology.orchestra.run.smithi112.stdout:3722 punch from 0x62df1d8 to 0x62ed436, (0xe25e bytes) 2024-09-04T21:54:49.956 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:49.963 INFO:teuthology.orchestra.run.smithi112.stdout:3725 trunc from 0xbe587fa to 0x6aeec31 2024-09-04T21:54:49.976 INFO:teuthology.orchestra.run.smithi112.stdout:3726 read 0x3dd6bf thru 0x3e053a (0x2e7c bytes) 2024-09-04T21:54:49.977 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:49.994 INFO:teuthology.orchestra.run.smithi112.stdout:3729 write 0xd77c1f6 thru 0xd781d9f (0x5baa bytes) 2024-09-04T21:54:50.008 INFO:teuthology.orchestra.run.smithi112.stdout:3731 trunc from 0xd781da0 to 0x7c6e234 2024-09-04T21:54:50.023 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:50.023 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:50.023 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:50.023 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:50.023 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:50.026 INFO:teuthology.orchestra.run.smithi112.stdout:3733 write 0x560fcc5 thru 0x561b25e (0xb59a bytes) 2024-09-04T21:54:50.035 INFO:teuthology.orchestra.run.smithi112.stdout:3736 read 0x4127dd thru 0x416581 (0x3da5 bytes) 2024-09-04T21:54:50.035 INFO:teuthology.orchestra.run.smithi112.stdout:3739 punch from 0x3c61a43 to 0x3c6a44d, (0x8a0a bytes) 2024-09-04T21:54:50.038 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:50.033+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:50.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:50.045 INFO:teuthology.orchestra.run.smithi112.stdout:3740 trunc from 0x7c6e234 to 0x295d59a 2024-09-04T21:54:50.081 INFO:teuthology.orchestra.run.smithi112.stdout:3744 writesame 0x6dc8f90 thru 0x6dce6d7 data_size 0x39(0x5748 bytes) 2024-09-04T21:54:50.084 INFO:teuthology.orchestra.run.smithi112.stdout:3745 trunc from 0x6dce6d8 to 0x91f64ed 2024-09-04T21:54:50.097 INFO:teuthology.orchestra.run.smithi112.stdout:3746 trunc from 0x91f64ed to 0x750ae32 2024-09-04T21:54:50.113 INFO:teuthology.orchestra.run.smithi112.stdout:3747 trunc from 0x750ae32 to 0x4b8fd3b 2024-09-04T21:54:50.126 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:50.127 INFO:teuthology.orchestra.run.smithi112.stdout:3750 trunc from 0x4b8fd3b to 0x42f52d5 2024-09-04T21:54:50.139 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:50.137+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:50.140 INFO:teuthology.orchestra.run.smithi112.stdout:3751 read 0x24eea17 thru 0x24fca0d (0xdff7 bytes) 2024-09-04T21:54:50.140 INFO:teuthology.orchestra.run.smithi112.stdout:3752 punch from 0x371068d to 0x371ffd0, (0xf943 bytes) 2024-09-04T21:54:50.158 INFO:teuthology.orchestra.run.smithi112.stdout:3753 write 0x7552f11 thru 0x75624a3 (0xf593 bytes) 2024-09-04T21:54:50.179 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:50.179 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:50.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:50.179 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.326534203999998, 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:50.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:50.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.186 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.200486941000001, 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:50.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:50.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:50.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.003933302999997, 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.193 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:50.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.976543804999999, 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:50.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:50.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:50.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.953279334000001, 2024-09-04T21:54:50.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:50.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:50.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.925521543999999, 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:50.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:50.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:50.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.925405982000001, 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.219 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:50.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:50.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.625231196999998, 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:50.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:50.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:50.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 43.466098645000002, 2024-09-04T21:54:50.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:50.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:50.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 43.465872587, 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.239 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:50.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:50.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.988659519000002, 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:50.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:50.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.759271408000004, 2024-09-04T21:54:50.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.461080766999999, 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.460967912000001, 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:50.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:50.272 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.451306944000002, 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:50.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:50.277 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.431457219000002, 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:50.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:50.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.281228532999997, 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:50.282 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:50.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.614901627999998, 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.611762442, 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:50.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.593451451999996, 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:50.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:50.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:50.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.305 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.305 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.305 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:50.306 INFO:teuthology.orchestra.run.smithi112.stdout:3754 trunc from 0x75624a4 to 0x306359d 2024-09-04T21:54:50.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:50.308 INFO:teuthology.orchestra.run.smithi112.stdout:3756 read 0xe41147 thru 0xe4fd36 (0xebf0 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3761 read 0x269b91f thru 0x26a2616 (0x6cf8 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3763 write 0x736bd50 thru 0x73776d1 (0xb982 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3766 read 0x6378ba2 thru 0x637c4cf (0x392e bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3767 write 0x55a8175 thru 0x55b4768 (0xc5f4 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3768 writesame 0x175c566 thru 0x1761487 data_size 0x7(0x4f22 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3769 read 0xaaa47d thru 0xab2574 (0x80f8 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3771 write 0x2631ead thru 0x263bc8c (0x9de0 bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3772 punch from 0x1edac15 to 0x1edd391, (0x277c bytes) 2024-09-04T21:54:50.309 INFO:teuthology.orchestra.run.smithi112.stdout:3774 read 0x528cd3 thru 0x52cb90 (0x3ebe bytes) 2024-09-04T21:54:50.310 INFO:teuthology.orchestra.run.smithi112.stdout:3776 write 0x8cb1d6b thru 0x8cbb8ee (0x9b84 bytes) 2024-09-04T21:54:50.310 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:50.310 INFO:teuthology.orchestra.run.smithi112.stdout:3779 write 0x2c5203 thru 0x2ca190 (0x4f8e bytes) 2024-09-04T21:54:50.310 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:50.301+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:50.334 INFO:teuthology.orchestra.run.smithi112.stdout:3780 write 0xb7d45f8 thru 0xb7de9dd (0xa3e6 bytes) 2024-09-04T21:54:50.407 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:50.405+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:50.457 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:50.457 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:50.457 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:50.458 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:50.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:50.508 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:50.505+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:50.587 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:50.587 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:50.588 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:50.588 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:50.588 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:50.601 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:50.645 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:50.645+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:50.710 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:50.705+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:50.729 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:50.729 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:50.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:50.729 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32654 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 4800512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.138803770999999, 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027786758000000002, 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.730 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32654 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588400+0000", 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3399999999999999e-07 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588403+0000", 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7249999999999999e-06 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588414+0000", 2024-09-04T21:54:50.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0559e-05 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588421+0000", 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5109999999999997e-06 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590927+0000", 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025057339999999999 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590933+0000", 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5820000000000001e-06 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610352+0000", 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.019419174000000001 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610442+0000", 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0125999999999997e-05 2024-09-04T21:54:50.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.616186+0000", 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057437130000000001 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32670 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5062656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.138189975, 2024-09-04T21:54:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024152731, 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32670 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:50.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4199999999999995e-07 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589017+0000", 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.224e-06 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589031+0000", 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4747e-05 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589039+0000", 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591456+0000", 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024168549999999999 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591462+0000", 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4979999999999997e-06 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613066+0000", 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0216037 2024-09-04T21:54:50.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613120+0000", 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4209999999999998e-05 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613165+0000", 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5763999999999998e-05 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32671 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5079040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.138150567, 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024282133000000001, 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.741 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32671 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589053+0000", 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589057+0000", 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:50.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589074+0000", 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7201999999999999e-05 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589083+0000", 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5739999999999996e-06 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591487+0000", 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024049150000000001 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591493+0000", 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-06 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613189+0000", 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021695945000000001 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613260+0000", 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0905e-05 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613334+0000", 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4425999999999995e-05 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32672 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5095424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.138107964, 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024400369000000002, 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32672 2024-09-04T21:54:50.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1299999999999999e-07 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589099+0000", 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.591e-06 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589115+0000", 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5591999999999999e-05 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589128+0000", 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3210000000000001e-05 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591520+0000", 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023920529999999999 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591526+0000", 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613358+0000", 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021832949000000001 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613427+0000", 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8955999999999995e-05 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613495+0000", 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7694999999999995e-05 2024-09-04T21:54:50.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32673 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.138063834, 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024527538000000002, 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32673 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589140+0000", 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589143+0000", 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.331e-06 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589153+0000", 2024-09-04T21:54:50.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7270000000000002e-06 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589161+0000", 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8149999999999995e-06 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591553+0000", 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023916469999999998 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591558+0000", 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1649999999999997e-06 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613520+0000", 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021961729999999999 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613591+0000", 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0735000000000006e-05 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613666+0000", 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5754999999999996e-05 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32674 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5128192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.138030342, 2024-09-04T21:54:50.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024653952999999999, 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32674 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589173+0000", 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000005e-07 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589186+0000", 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2972999999999999e-05 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589204+0000", 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7666999999999999e-05 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589212+0000", 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3999999999999992e-06 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591589+0000", 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023771370000000001 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591594+0000", 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3149999999999999e-06 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613690+0000", 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022095789000000001 2024-09-04T21:54:50.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613759+0000", 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9142999999999999e-05 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613826+0000", 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6958999999999994e-05 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32675 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5144576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.137977923000001, 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024775136, 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.763 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32675 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589226+0000", 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0500000000000003e-07 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589229+0000", 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999998e-06 2024-09-04T21:54:50.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589239+0000", 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589247+0000", 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0430000000000008e-06 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591622+0000", 2024-09-04T21:54:50.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023745099999999998 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591627+0000", 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613851+0000", 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022223613 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613923+0000", 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2203e-05 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614000+0000", 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32676 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5160960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.768 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.137944686000001, 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024914054000000001, 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32676 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589259+0000", 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589276+0000", 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7686e-05 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589288+0000", 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1234e-05 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589296+0000", 2024-09-04T21:54:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5769999999999994e-06 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591654+0000", 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023581769999999999 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591660+0000", 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3109999999999996e-06 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614025+0000", 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022365051 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614099+0000", 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3881000000000006e-05 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614172+0000", 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3498999999999995e-05 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32677 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5177344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.137886654999999, 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025050084, 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.774 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32677 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589317+0000", 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5300000000000003e-07 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589329+0000", 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1951e-05 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589346+0000", 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7108e-05 2024-09-04T21:54:50.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589354+0000", 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2630000000000003e-06 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591674+0000", 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023197980000000001 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591677+0000", 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5909999999999998e-06 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614205+0000", 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022528256999999999 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614287+0000", 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1886999999999997e-05 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614366+0000", 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9475999999999996e-05 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32678 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5193728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.137834463000001, 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025172158, 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32678 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589369+0000", 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8400000000000004e-07 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589373+0000", 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589391+0000", 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8077e-05 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589400+0000", 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0790000000000007e-06 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591689+0000", 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002288815 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591692+0000", 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.666e-06 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614392+0000", 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022700287 2024-09-04T21:54:50.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614462+0000", 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0468000000000004e-05 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614540+0000", 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8104999999999998e-05 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32679 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5210112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.13779053, 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025302998, 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32679 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.785 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589413+0000", 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0100000000000004e-07 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589417+0000", 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7689999999999998e-06 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589435+0000", 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8185e-05 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589444+0000", 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4029999999999996e-06 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591704+0000", 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022601729999999999 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591708+0000", 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.067e-06 2024-09-04T21:54:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614571+0000", 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022863270000000002 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614640+0000", 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909e-05 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614715+0000", 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5520999999999997e-05 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32680 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5226496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.137744919999999, 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025450210000000001, 2024-09-04T21:54:50.790 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32680 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589459+0000", 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589462+0000", 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7780000000000001e-06 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589479+0000", 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6651e-05 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589488+0000", 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6309999999999992e-06 2024-09-04T21:54:50.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591737+0000", 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002249301 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591742+0000", 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4790000000000004e-06 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614747+0000", 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.023004539000000001 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614825+0000", 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8566000000000006e-05 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614908+0000", 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2558999999999993e-05 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32929 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5120000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.128129055999999, 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024146612000000001, 2024-09-04T21:54:50.796 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32929 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:50.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599077+0000", 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.373e-06 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599083+0000", 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5060000000000001e-06 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599088+0000", 2024-09-04T21:54:50.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0690000000000001e-06 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601461+0000", 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023728299999999998 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601465+0000", 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5129999999999998e-06 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623133+0000", 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021667782 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623178+0000", 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4950000000000002e-05 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623220+0000", 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2076999999999997e-05 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.801 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32930 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5136384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.128107331999999, 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024238576000000001, 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.802 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32930 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599096+0000", 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599098+0000", 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2979999999999999e-06 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599105+0000", 2024-09-04T21:54:50.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5479999999999998e-06 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599110+0000", 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1109999999999999e-06 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601491+0000", 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023813800000000002 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601496+0000", 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623247+0000", 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021751177 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623291+0000", 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4184999999999998e-05 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623334+0000", 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2970999999999999e-05 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32931 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5152768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.128085066000001, 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024330243000000001, 2024-09-04T21:54:50.807 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32931 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.61e-07 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599121+0000", 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.683e-06 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599129+0000", 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6359999999999994e-06 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599134+0000", 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5049999999999998e-06 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601519+0000", 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002384546 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601521+0000", 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2919999999999998e-06 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623360+0000", 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021839164000000001 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623405+0000", 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4243999999999999e-05 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623448+0000", 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3411999999999997e-05 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32932 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5169152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.128060032, 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024418507999999998, 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32932 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599144+0000", 2024-09-04T21:54:50.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4799999999999997e-07 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599153+0000", 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9930000000000001e-06 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599161+0000", 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0900000000000005e-06 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599167+0000", 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601543+0000", 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023758540000000002 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601547+0000", 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1019999999999999e-06 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623474+0000", 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021927782 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623519+0000", 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4263e-05 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623561+0000", 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2483e-05 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.817 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32933 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5185536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.128025937, 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024496766999999999, 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.818 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32933 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-07 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599180+0000", 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599188+0000", 2024-09-04T21:54:50.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599203+0000", 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5099999999999999e-05 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601573+0000", 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002369566 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601577+0000", 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5090000000000004e-06 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623587+0000", 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022009635999999999 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623631+0000", 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4605999999999999e-05 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623674+0000", 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2274000000000002e-05 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32934 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5201920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.127989764999999, 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024572765999999999, 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32934 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599214+0000", 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599217+0000", 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.095e-06 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599224+0000", 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6760000000000007e-06 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599230+0000", 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8039999999999998e-06 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601603+0000", 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023727090000000002 2024-09-04T21:54:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601608+0000", 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-06 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623700+0000", 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022092219999999999 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623744+0000", 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4013e-05 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623786+0000", 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1749000000000003e-05 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32935 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5218304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.127963480999998, 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024658065999999999, 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32935 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599240+0000", 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3500000000000002e-07 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599249+0000", 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599258+0000", 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7069999999999998e-06 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599264+0000", 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4790000000000001e-06 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601634+0000", 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023694020000000001 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601638+0000", 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623812+0000", 2024-09-04T21:54:50.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022173986 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623856+0000", 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3696000000000003e-05 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623897+0000", 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1530999999999999e-05 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32936 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5234688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.127928130000001, 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024740649, 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32936 2024-09-04T21:54:50.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2099999999999997e-07 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599278+0000", 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8870000000000001e-06 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599285+0000", 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4449999999999999e-06 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599291+0000", 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601668+0000", 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023769379999999999 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601673+0000", 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8210000000000001e-06 2024-09-04T21:54:50.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623927+0000", 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022253536000000001 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623972+0000", 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5216999999999998e-05 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.624015+0000", 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3519000000000001e-05 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:50.840 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:50.840 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:50.842 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:50.843 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:50.841+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:50.943 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:50.941+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:51.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:51.042 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:51.043 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:51.044 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:51.041+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:51.061 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:51.061 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:51.076 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:51.076 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:51.093 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:51.094 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:51.115 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:51.115 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:51.135 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:51.136 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:51.145 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:51.141+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:51.156 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:51.156 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:51.157 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:51.158 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:51.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:51.178 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:51.178 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:51.246 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:51.245+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:51.293 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:51.294 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:51.307 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:51.355 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:51.353+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:51.356 INFO:teuthology.orchestra.run.smithi112.stdout:3781 writesame 0xa598a02 thru 0xa59a2aa data_size 0x1(0x18a9 bytes) 2024-09-04T21:54:51.363 INFO:teuthology.orchestra.run.smithi112.stdout:3782 write 0x36b380c thru 0x36b6ffb (0x37f0 bytes) 2024-09-04T21:54:51.377 INFO:teuthology.orchestra.run.smithi112.stdout:3786 trunc from 0xb7de9de to 0x6939281 2024-09-04T21:54:51.395 INFO:teuthology.orchestra.run.smithi112.stdout:3788 write 0x43563ca thru 0x435d988 (0x75bf bytes) 2024-09-04T21:54:51.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:54:51.410 INFO:teuthology.orchestra.run.smithi112.stdout:3789 writesame 0x6ff42de thru 0x6ffe21a data_size 0x1f(0x9f3d bytes) 2024-09-04T21:54:51.418 INFO:teuthology.orchestra.run.smithi112.stdout:3790 trunc from 0x6ffe21b to 0xc69bbd6 2024-09-04T21:54:51.437 INFO:teuthology.orchestra.run.smithi112.stdout:3791 trunc from 0xc69bbd6 to 0x54e059d 2024-09-04T21:54:51.447 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:51.445+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:51.454 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:51.454 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:51.465 INFO:teuthology.orchestra.run.smithi112.stdout:3792 writesame 0xa4d4750 thru 0xa4d9a68 data_size 0x7(0x5319 bytes) 2024-09-04T21:54:51.465 INFO:teuthology.orchestra.run.smithi112.stdout:3793 write 0x58fa7ed thru 0x590a41d (0xfc31 bytes) 2024-09-04T21:54:51.470 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:51.477 INFO:teuthology.orchestra.run.smithi112.stdout:3794 punch from 0x7fb81ae to 0x7fc4028, (0xbe7a bytes) 2024-09-04T21:54:51.478 INFO:teuthology.orchestra.run.smithi112.stdout:3795 read 0x201abf7 thru 0x20269de (0xbde8 bytes) 2024-09-04T21:54:51.478 INFO:teuthology.orchestra.run.smithi112.stdout:3796 write 0x6412548 thru 0x6422017 (0xfad0 bytes) 2024-09-04T21:54:51.487 INFO:teuthology.orchestra.run.smithi112.stdout:3798 write 0x71f143d thru 0x71f3d20 (0x28e4 bytes) 2024-09-04T21:54:51.496 INFO:teuthology.orchestra.run.smithi112.stdout:3799 trunc from 0xa4d9a69 to 0x75c1649 2024-09-04T21:54:51.504 INFO:teuthology.orchestra.run.smithi112.stdout:3802 read 0x27676ef thru 0x276f474 (0x7d86 bytes) 2024-09-04T21:54:51.515 INFO:teuthology.orchestra.run.smithi112.stdout:3803 writesame 0x9647562 thru 0x9650035 data_size 0xa(0x8ad4 bytes) 2024-09-04T21:54:51.516 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:51.516 INFO:teuthology.orchestra.run.smithi112.stdout:3807 punch from 0x785fc57 to 0x78645b8, (0x4961 bytes) 2024-09-04T21:54:51.533 INFO:teuthology.orchestra.run.smithi112.stdout:3808 writesame 0xeb9a34b thru 0xeba317b data_size 0x1(0x8e31 bytes) 2024-09-04T21:54:51.533 INFO:teuthology.orchestra.run.smithi112.stdout:3810 read 0x89fb9de thru 0x89fd0d0 (0x16f3 bytes) 2024-09-04T21:54:51.534 INFO:teuthology.orchestra.run.smithi112.stdout:3811 writesame 0x2e7a636 thru 0x2e7b6d6 data_size 0x21(0x10a1 bytes) 2024-09-04T21:54:51.538 INFO:teuthology.orchestra.run.smithi112.stdout:3812 read 0xdc1bed8 thru 0xdc1da6f (0x1b98 bytes) 2024-09-04T21:54:51.552 INFO:teuthology.orchestra.run.smithi112.stdout:3814 trunc from 0xeba317c to 0x54dfb4e 2024-09-04T21:54:51.555 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:51.553+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:51.580 INFO:teuthology.orchestra.run.smithi112.stdout:3818 writesame 0x8290037 thru 0x8299cd1 data_size 0x1(0x9c9b bytes) 2024-09-04T21:54:51.581 INFO:teuthology.orchestra.run.smithi112.stdout:3819 read 0x5985d55 thru 0x5992664 (0xc910 bytes) 2024-09-04T21:54:51.581 INFO:teuthology.orchestra.run.smithi112.stdout:3821 read 0x5119b36 thru 0x511bd67 (0x2232 bytes) 2024-09-04T21:54:51.581 INFO:teuthology.orchestra.run.smithi112.stdout:3822 write 0x4239c39 thru 0x42413df (0x77a7 bytes) 2024-09-04T21:54:51.583 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:51.583 INFO:teuthology.orchestra.run.smithi112.stdout:3825 punch from 0x237fd9f to 0x238b845, (0xbaa6 bytes) 2024-09-04T21:54:51.591 INFO:teuthology.orchestra.run.smithi112.stdout:3826 write 0x9799bf3 thru 0x979be6f (0x227d bytes) 2024-09-04T21:54:51.594 INFO:teuthology.orchestra.run.smithi112.stdout:3828 write 0x1ffc463 thru 0x2008ab0 (0xc64e bytes) 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:3831 punch from 0x8f70cab to 0x8f7f7a5, (0xeafa bytes) 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:3833 punch from 0x3f510f1 to 0x3f52a81, (0x1990 bytes) 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:3835 read 0x55fe352 thru 0x5604b7e (0x682d bytes) 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:3836 read 0x783e0c6 thru 0x7846fcb (0x8f06 bytes) 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:3839 read 0x6e16898 thru 0x6e1dd90 (0x74f9 bytes) 2024-09-04T21:54:51.600 INFO:teuthology.orchestra.run.smithi112.stdout:3840 punch from 0x43d4ea5 to 0x43d7467, (0x25c2 bytes) 2024-09-04T21:54:51.601 INFO:teuthology.orchestra.run.smithi112.stdout:3841 write 0x1ec8a6f thru 0x1ed2cfb (0xa28d bytes) 2024-09-04T21:54:51.602 INFO:teuthology.orchestra.run.smithi112.stdout:3842 write 0x50544c5 thru 0x505e0db (0x9c17 bytes) 2024-09-04T21:54:51.616 INFO:teuthology.orchestra.run.smithi112.stdout:3847 write 0xa38a73e thru 0xa38ca61 (0x2324 bytes) 2024-09-04T21:54:51.622 INFO:teuthology.orchestra.run.smithi112.stdout:3848 read 0x76f0f0b thru 0x76fc8c6 (0xb9bc bytes) 2024-09-04T21:54:51.623 INFO:teuthology.orchestra.run.smithi112.stdout:3849 writesame 0x17b3044 thru 0x17c27a6 data_size 0x1(0xf763 bytes) 2024-09-04T21:54:51.649 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:51.649+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:51.650 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:51.650 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:51.657 INFO:teuthology.orchestra.run.smithi112.stdout:3850 trunc from 0xa38ca62 to 0xb805c36 2024-09-04T21:54:51.667 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:51.683 INFO:teuthology.orchestra.run.smithi112.stdout:3851 trunc from 0xb805c36 to 0x4f8f00c 2024-09-04T21:54:51.715 INFO:teuthology.orchestra.run.smithi112.stdout:3852 trunc from 0x4f8f00c to 0x3b8195 2024-09-04T21:54:51.744 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:51.744 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:51.755 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:51.753+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:51.764 INFO:teuthology.orchestra.run.smithi112.stdout:3856 trunc from 0x3b8195 to 0x8c21999 2024-09-04T21:54:51.790 INFO:teuthology.orchestra.run.smithi112.stdout:3859 write 0xb1d28ec thru 0xb1e11a3 (0xe8b8 bytes) 2024-09-04T21:54:51.803 INFO:teuthology.orchestra.run.smithi112.stdout:3860 write 0x65a3aa6 thru 0x65a6120 (0x267b bytes) 2024-09-04T21:54:51.811 INFO:teuthology.orchestra.run.smithi112.stdout:3861 write 0x824eb70 thru 0x82569ac (0x7e3d bytes) 2024-09-04T21:54:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:3862 punch from 0x2e9f5f8 to 0x2e9fc06, (0x60e bytes) 2024-09-04T21:54:51.841 INFO:teuthology.orchestra.run.smithi112.stdout:3863 write 0xeb00ebe thru 0xeb0813c (0x727f bytes) 2024-09-04T21:54:51.847 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:51.847 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:51.851 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:51.849+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:51.853 INFO:teuthology.orchestra.run.smithi112.stdout:3865 write 0xb38da6e thru 0xb391c11 (0x41a4 bytes) 2024-09-04T21:54:51.857 INFO:teuthology.orchestra.run.smithi112.stdout:3866 write 0x190e621 thru 0x1917cc0 (0x96a0 bytes) 2024-09-04T21:54:51.864 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:51.890 INFO:teuthology.orchestra.run.smithi112.stdout:3868 trunc from 0xeb0813d to 0x7797e81 2024-09-04T21:54:51.932 INFO:teuthology.orchestra.run.smithi112.stdout:3871 writesame 0x8a660b2 thru 0x8a6a6a8 data_size 0x1(0x45f7 bytes) 2024-09-04T21:54:51.950 INFO:teuthology.orchestra.run.smithi112.stdout:3872 writesame 0xb851cfd thru 0xb85f85d data_size 0x47(0xdb61 bytes) 2024-09-04T21:54:51.950 INFO:teuthology.orchestra.run.smithi112.stdout:3873 read 0xa57f617 thru 0xa58db9b (0xe585 bytes) 2024-09-04T21:54:51.950 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:51.950 INFO:teuthology.orchestra.run.smithi112.stdout:3875 punch from 0x9fe20c3 to 0x9ff0ed5, (0xee12 bytes) 2024-09-04T21:54:51.952 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:51.949+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:51.969 INFO:teuthology.orchestra.run.smithi112.stdout:3876 write 0xe86356f thru 0xe86a3c1 (0x6e53 bytes) 2024-09-04T21:54:51.978 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.004 INFO:teuthology.orchestra.run.smithi112.stdout:3879 trunc from 0xe86a3c2 to 0x48e39d4 2024-09-04T21:54:52.032 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.042 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:52.042 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:52.051 INFO:teuthology.orchestra.run.smithi112.stdout:3884 write 0x63e2ac8 thru 0x63ead65 (0x829e bytes) 2024-09-04T21:54:52.052 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:52.049+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:52.057 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:52.061 INFO:teuthology.orchestra.run.smithi112.stdout:3885 punch from 0x317687 to 0x324ff0, (0xd969 bytes) 2024-09-04T21:54:52.061 INFO:teuthology.orchestra.run.smithi112.stdout:3886 writesame 0x2601dd thru 0x26d47f data_size 0x1(0xd2a3 bytes) 2024-09-04T21:54:52.093 INFO:teuthology.orchestra.run.smithi112.stdout:3887 write 0x38f0eb4 thru 0x38f2d49 (0x1e96 bytes) 2024-09-04T21:54:52.107 INFO:teuthology.orchestra.run.smithi112.stdout:3891 trunc from 0x63ead66 to 0x3218571 2024-09-04T21:54:52.153 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:52.149+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:52.171 INFO:teuthology.orchestra.run.smithi112.stdout:3892 write 0xce3b45a thru 0xce48536 (0xd0dd bytes) 2024-09-04T21:54:52.179 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:52.179 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:52.184 INFO:teuthology.orchestra.run.smithi112.stdout:3893 punch from 0x4b50895 to 0x4b55239, (0x49a4 bytes) 2024-09-04T21:54:52.184 INFO:teuthology.orchestra.run.smithi112.stdout:3894 writesame 0x92edd26 thru 0x92fd790 data_size 0x33(0xfa6b bytes) 2024-09-04T21:54:52.197 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:52.198 INFO:teuthology.orchestra.run.smithi112.stdout:3895 read 0x724d646 thru 0x725a0de (0xca99 bytes) 2024-09-04T21:54:52.198 INFO:teuthology.orchestra.run.smithi112.stdout:3896 write 0x4bfa5ba thru 0x4bfc2b7 (0x1cfe bytes) 2024-09-04T21:54:52.204 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.206 INFO:teuthology.orchestra.run.smithi112.stdout:3899 trunc from 0xce48537 to 0xddbbd9f 2024-09-04T21:54:52.217 INFO:teuthology.orchestra.run.smithi112.stdout:3900 writesame 0x2dab382 thru 0x2db85b7 data_size 0x3(0xd236 bytes) 2024-09-04T21:54:52.242 INFO:teuthology.orchestra.run.smithi112.stdout:3901 writesame 0xae3ff18 thru 0xae4fc8e data_size 0x1(0xfd77 bytes) 2024-09-04T21:54:52.254 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:52.253+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:52.274 INFO:teuthology.orchestra.run.smithi112.stdout:3903 read 0x930cf47 thru 0x9319746 (0xc800 bytes) 2024-09-04T21:54:52.274 INFO:teuthology.orchestra.run.smithi112.stdout:3906 read 0x818a452 thru 0x818c3e5 (0x1f94 bytes) 2024-09-04T21:54:52.274 INFO:teuthology.orchestra.run.smithi112.stdout:3907 read 0x6a8d21f thru 0x6a9b2c5 (0xe0a7 bytes) 2024-09-04T21:54:52.274 INFO:teuthology.orchestra.run.smithi112.stdout:3911 read 0x534709a thru 0x534d8da (0x6841 bytes) 2024-09-04T21:54:52.291 INFO:teuthology.orchestra.run.smithi112.stdout:3913 write 0xed0015b thru 0xed03691 (0x3537 bytes) 2024-09-04T21:54:52.303 INFO:teuthology.orchestra.run.smithi112.stdout:3914 trunc from 0xed03692 to 0xc75673b 2024-09-04T21:54:52.326 INFO:teuthology.orchestra.run.smithi112.stdout:3915 write 0x76fa58f thru 0x7707fe7 (0xda59 bytes) 2024-09-04T21:54:52.342 INFO:teuthology.orchestra.run.smithi112.stdout:3916 read 0x32aa64e thru 0x32b6e47 (0xc7fa bytes) 2024-09-04T21:54:52.342 INFO:teuthology.orchestra.run.smithi112.stdout:3917 read 0x57b10ad thru 0x57b942a (0x837e bytes) 2024-09-04T21:54:52.342 INFO:teuthology.orchestra.run.smithi112.stdout:3920 writesame 0x3daeb4f thru 0x3db09c9 data_size 0x1b(0x1e7b bytes) 2024-09-04T21:54:52.350 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.350 INFO:teuthology.orchestra.run.smithi112.stdout:3922 read 0x8da313 thru 0x8dbb0b (0x17f9 bytes) 2024-09-04T21:54:52.350 INFO:teuthology.orchestra.run.smithi112.stdout:3923 read 0x798ddc7 thru 0x79968f7 (0x8b31 bytes) 2024-09-04T21:54:52.350 INFO:teuthology.orchestra.run.smithi112.stdout:3924 punch from 0x66cc234 to 0x66d3fea, (0x7db6 bytes) 2024-09-04T21:54:52.350 INFO:teuthology.orchestra.run.smithi112.stdout:3925 writesame 0x1f66cd9 thru 0x1f6a63f data_size 0x1(0x3967 bytes) 2024-09-04T21:54:52.355 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:52.353+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:52.373 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:54:52.374 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.374 INFO:teuthology.orchestra.run.smithi112.stdout:3927 write 0x94ea8de thru 0x94f4c31 (0xa354 bytes) 2024-09-04T21:54:52.377 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:52.377 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:52.385 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7] out_osds: [1, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:54:52.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:54:52.386 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-04T21:54:52.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-04T21:54:52.387 INFO:teuthology.orchestra.run.smithi112.stdout:3928 punch from 0x75e0953 to 0x75e848d, (0x7b3a bytes) 2024-09-04T21:54:52.387 INFO:teuthology.orchestra.run.smithi112.stdout:3929 read 0x36b3ba thru 0x37a9ae (0xf5f5 bytes) 2024-09-04T21:54:52.387 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.387 INFO:teuthology.orchestra.run.smithi112.stdout:3934 writesame 0x8f9de94 thru 0x8fa36fc data_size 0x1(0x5869 bytes) 2024-09-04T21:54:52.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:52.404 INFO:teuthology.orchestra.run.smithi112.stdout:3936 write 0x9fa13b3 thru 0x9facbb4 (0xb802 bytes) 2024-09-04T21:54:52.419 INFO:teuthology.orchestra.run.smithi112.stdout:3937 write 0xcabea68 thru 0xcac1048 (0x25e1 bytes) 2024-09-04T21:54:52.425 INFO:teuthology.orchestra.run.smithi112.stdout:3940 punch from 0x32a0c3d to 0x32ad2f6, (0xc6b9 bytes) 2024-09-04T21:54:52.425 INFO:teuthology.orchestra.run.smithi112.stdout:3942 punch from 0x38365be to 0x38398d8, (0x331a bytes) 2024-09-04T21:54:52.426 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.426 INFO:teuthology.orchestra.run.smithi112.stdout:3946 write 0x169d2c6 thru 0x16a4cda (0x7a15 bytes) 2024-09-04T21:54:52.434 INFO:teuthology.orchestra.run.smithi112.stdout:3947 write 0x16354da thru 0x163a093 (0x4bba bytes) 2024-09-04T21:54:52.441 INFO:teuthology.orchestra.run.smithi112.stdout:3948 read 0x9672c34 thru 0x9680df5 (0xe1c2 bytes) 2024-09-04T21:54:52.441 INFO:teuthology.orchestra.run.smithi112.stdout:3949 punch from 0xbb4bffd to 0xbb56883, (0xa886 bytes) 2024-09-04T21:54:52.441 INFO:teuthology.orchestra.run.smithi112.stdout:3950 read 0x3dbe7e7 thru 0x3dc0cf9 (0x2513 bytes) 2024-09-04T21:54:52.442 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.442 INFO:teuthology.orchestra.run.smithi112.stdout:3953 write 0x4e153ab thru 0x4e1e4e0 (0x9136 bytes) 2024-09-04T21:54:52.455 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:52.453+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:52.456 INFO:teuthology.orchestra.run.smithi112.stdout:3954 trunc from 0xcac1049 to 0xa2fe97b 2024-09-04T21:54:52.481 INFO:teuthology.orchestra.run.smithi112.stdout:3955 write 0xa89568c thru 0xa897801 (0x2176 bytes) 2024-09-04T21:54:52.488 INFO:teuthology.orchestra.run.smithi112.stdout:3956 trunc from 0xa897802 to 0x9948f08 2024-09-04T21:54:52.504 INFO:teuthology.orchestra.run.smithi112.stdout:3957 punch from 0x7412a0e to 0x741dc76, (0xb268 bytes) 2024-09-04T21:54:52.516 INFO:teuthology.orchestra.run.smithi112.stdout:3958 write 0xaec361d thru 0xaecf8d5 (0xc2b9 bytes) 2024-09-04T21:54:52.523 INFO:teuthology.orchestra.run.smithi112.stdout:3960 read 0x270bb7c thru 0x2714266 (0x86eb bytes) 2024-09-04T21:54:52.523 INFO:teuthology.orchestra.run.smithi112.stdout:3961 read 0x3844665 thru 0x384ba82 (0x741e bytes) 2024-09-04T21:54:52.523 INFO:teuthology.orchestra.run.smithi112.stdout:3962 read 0x8f88753 thru 0x8f8cb8d (0x443b bytes) 2024-09-04T21:54:52.523 INFO:teuthology.orchestra.run.smithi112.stdout:3963 punch from 0x4b2e98d to 0x4b391f6, (0xa869 bytes) 2024-09-04T21:54:52.524 INFO:teuthology.orchestra.run.smithi112.stdout:3964 punch from 0x7f4d804 to 0x7f503c1, (0x2bbd bytes) 2024-09-04T21:54:52.524 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.529 INFO:teuthology.orchestra.run.smithi112.stdout:3966 write 0xb4eca2a thru 0xb4f3f62 (0x7539 bytes) 2024-09-04T21:54:52.532 INFO:teuthology.orchestra.run.smithi112.stdout:3967 read 0x5744cb9 thru 0x574d41e (0x8766 bytes) 2024-09-04T21:54:52.533 INFO:teuthology.orchestra.run.smithi112.stdout:3968 trunc from 0xb4f3f63 to 0xb26a07e 2024-09-04T21:54:52.539 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:52.539 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:52.543 INFO:teuthology.orchestra.run.smithi112.stdout:3969 read 0x5710643 thru 0x571fa1f (0xf3dd bytes) 2024-09-04T21:54:52.543 INFO:teuthology.orchestra.run.smithi112.stdout:3971 read 0x90f3c40 thru 0x90f525a (0x161b bytes) 2024-09-04T21:54:52.543 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.543 INFO:teuthology.orchestra.run.smithi112.stdout:3975 punch from 0xe2d23b to 0xe358a4, (0x8669 bytes) 2024-09-04T21:54:52.551 INFO:teuthology.orchestra.run.smithi112.stdout:3977 write 0xc445664 thru 0xc44a1a8 (0x4b45 bytes) 2024-09-04T21:54:52.556 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:52.556 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:52.553+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:52.557 INFO:teuthology.orchestra.run.smithi112.stdout:3978 write 0x4d3a537 thru 0x4d3bb43 (0x160d bytes) 2024-09-04T21:54:52.559 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.559 INFO:teuthology.orchestra.run.smithi112.stdout:3980 write 0x72ec5d0 thru 0x72fc3f2 (0xfe23 bytes) 2024-09-04T21:54:52.568 INFO:teuthology.orchestra.run.smithi112.stdout:3981 writesame 0x1a9d988 thru 0x1aa6580 data_size 0x1(0x8bf9 bytes) 2024-09-04T21:54:52.582 INFO:teuthology.orchestra.run.smithi112.stdout:3982 read 0x26b1eb2 thru 0x26c0416 (0xe565 bytes) 2024-09-04T21:54:52.582 INFO:teuthology.orchestra.run.smithi112.stdout:3983 writesame 0x19f3460 thru 0x19fef0b data_size 0x1a(0xbaac bytes) 2024-09-04T21:54:52.588 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.588 INFO:teuthology.orchestra.run.smithi112.stdout:3985 punch from 0x7147ec3 to 0x7152f68, (0xb0a5 bytes) 2024-09-04T21:54:52.588 INFO:teuthology.orchestra.run.smithi112.stdout:3989 write 0x6a6ffe6 thru 0x6a71a39 (0x1a54 bytes) 2024-09-04T21:54:52.592 INFO:teuthology.orchestra.run.smithi112.stdout:3990 punch from 0xbc16909 to 0xbc20af6, (0xa1ed bytes) 2024-09-04T21:54:52.592 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.592 INFO:teuthology.orchestra.run.smithi112.stdout:3993 writesame 0x623840e thru 0x623db74 data_size 0x19(0x5767 bytes) 2024-09-04T21:54:52.597 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:52.597 INFO:teuthology.orchestra.run.smithi112.stdout:3995 clone 20 order 20 su 65536 sc 2 2024-09-04T21:54:52.657 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:52.653+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:52.713 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:52.714 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:52.717 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:54:52.717 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:52.717 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 69, 2024-09-04T21:54:52.717 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "7c7b20bb-5983-49a4-8b2c-296a55ea8d95", 2024-09-04T21:54:52.717 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-04T21:53:25.631463+0000", 2024-09-04T21:54:52.717 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-04T21:54:52.354699+0000", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "last_up_change": "2024-09-04T21:54:30.565194+0000", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "last_in_change": "2024-09-04T21:54:47.345059+0000", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_num": 5799936, 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_set": [ 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "pglog_hardlimit", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "purged_snapdirs", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_deletes", 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "sortbitwise" 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_version": 6, 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "full_ratio": 0.94999998807907104, 2024-09-04T21:54:52.718 INFO:teuthology.orchestra.run.smithi112.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_snapshot": "", 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_max": 3, 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "max_osd": 8, 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "require_min_compat_client": "luminous", 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "min_compat_client": "jewel", 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "require_osd_release": "squid", 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "allow_crimson": false, 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pools": [ 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 1, 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_name": ".mgr", 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "create_time": "2024-09-04T21:53:36.459454+0000", 2024-09-04T21:54:52.719 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 1, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_names": "hashpspool", 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "type": 1, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 2, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "min_size": 1, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_rule": 0, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_count": 0, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_target": 0, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "is_stretch_pool": false, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "object_hash": 2, 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_autoscale_mode": "off", 2024-09-04T21:54:52.720 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num": 1, 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num": 1, 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num_target": 1, 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_target": 1, 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_pending": 1, 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "last_pg_merge_meta": { 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "source_pgid": "0.0", 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "ready_epoch": 0, 2024-09-04T21:54:52.721 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_started": 0, 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_clean": 0, 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "source_version": "0'0", 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "target_version": "0'0" 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "last_change": "17", 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend": "0", 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "auid": 0, 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_mode": "selfmanaged", 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_seq": 0, 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_epoch": 0, 2024-09-04T21:54:52.722 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_snaps": [], 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps": "[]", 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_bytes": 0, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_objects": 0, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "tiers": [], 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_of": -1, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "read_tier": -1, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "write_tier": -1, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_mode": "none", 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_bytes": 0, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_objects": 0, 2024-09-04T21:54:52.723 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_flush_age": 0, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_evict_age": 0, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profile": "", 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_params": { 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "none" 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_period": 0, 2024-09-04T21:54:52.724 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_count": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "use_gmt_hitset": true, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "min_read_recency_for_promote": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "min_write_recency_for_promote": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_search_last_n": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "grade_table": [], 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "stripe_width": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "expected_num_objects": 0, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "fast_read": false, 2024-09-04T21:54:52.725 INFO:teuthology.orchestra.run.smithi112.stdout: "options": { 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_max": 32, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_min": 1 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "application_metadata": { 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": {} 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "read_balance": { 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "score_type": "Fair distribution", 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "score_acting": 6.9000000953674316, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "score_stable": 6.9000000953674316, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "optimal_score": 0.28999999165534973, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_acting": 2, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_stable": 2, 2024-09-04T21:54:52.726 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity_weighted": 0.875, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity": 1, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 2, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_name": "rbd", 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "create_time": "2024-09-04T21:53:43.849962+0000", 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 8193, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "type": 1, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 2, 2024-09-04T21:54:52.727 INFO:teuthology.orchestra.run.smithi112.stdout: "min_size": 1, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_rule": 0, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_count": 0, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_target": 0, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "is_stretch_pool": false, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "object_hash": 2, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_autoscale_mode": "off", 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num": 8, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num": 8, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num_target": 8, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_target": 8, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_pending": 8, 2024-09-04T21:54:52.728 INFO:teuthology.orchestra.run.smithi112.stdout: "last_pg_merge_meta": { 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "source_pgid": "0.0", 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "ready_epoch": 0, 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_started": 0, 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_clean": 0, 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "source_version": "0'0", 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "target_version": "0'0" 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_change": "21", 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend": "0", 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T21:54:52.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "auid": 0, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_mode": "selfmanaged", 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_seq": 2, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_epoch": 21, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_snaps": [], 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps": "[]", 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_bytes": 0, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_objects": 0, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "tiers": [], 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_of": -1, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "read_tier": -1, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "write_tier": -1, 2024-09-04T21:54:52.730 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_mode": "none", 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_bytes": 0, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_objects": 0, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_flush_age": 0, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_evict_age": 0, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profile": "", 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_params": { 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "none" 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_period": 0, 2024-09-04T21:54:52.731 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_count": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "use_gmt_hitset": true, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "min_read_recency_for_promote": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "min_write_recency_for_promote": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_search_last_n": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "grade_table": [], 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "stripe_width": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "expected_num_objects": 0, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "fast_read": false, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "options": {}, 2024-09-04T21:54:52.732 INFO:teuthology.orchestra.run.smithi112.stdout: "application_metadata": { 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd": {} 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "read_balance": { 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "score_type": "Fair distribution", 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "score_acting": 2.619999885559082, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "score_stable": 2.619999885559082, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "optimal_score": 0.86000001430511475, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_acting": 2.25, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_stable": 2.25, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity_weighted": 0.875, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity": 1, 2024-09-04T21:54:52.733 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 3, 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_name": "pool_client.0", 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: "create_time": "2024-09-04T21:53:54.481634+0000", 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 8193, 2024-09-04T21:54:52.734 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "type": 1, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 2, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "min_size": 1, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_rule": 0, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_count": 0, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_target": 0, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "is_stretch_pool": false, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "object_hash": 2, 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_autoscale_mode": "off", 2024-09-04T21:54:52.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num": 32, 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num": 32, 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num_target": 32, 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_target": 32, 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_pending": 32, 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "last_pg_merge_meta": { 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "source_pgid": "0.0", 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "ready_epoch": 0, 2024-09-04T21:54:52.736 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_started": 0, 2024-09-04T21:54:52.737 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_clean": 0, 2024-09-04T21:54:52.737 INFO:teuthology.orchestra.run.smithi112.stdout: "source_version": "0'0", 2024-09-04T21:54:52.737 INFO:teuthology.orchestra.run.smithi112.stdout: "target_version": "0'0" 2024-09-04T21:54:52.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.737 INFO:teuthology.orchestra.run.smithi112.stdout: "last_change": "64", 2024-09-04T21:54:52.738 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend": "0", 2024-09-04T21:54:52.738 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T21:54:52.738 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T21:54:52.738 INFO:teuthology.orchestra.run.smithi112.stdout: "auid": 0, 2024-09-04T21:54:52.738 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_mode": "selfmanaged", 2024-09-04T21:54:52.739 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_seq": 21, 2024-09-04T21:54:52.739 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_epoch": 64, 2024-09-04T21:54:52.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_snaps": [], 2024-09-04T21:54:52.739 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps": "[]", 2024-09-04T21:54:52.739 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_bytes": 0, 2024-09-04T21:54:52.739 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_objects": 0, 2024-09-04T21:54:52.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tiers": [], 2024-09-04T21:54:52.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_of": -1, 2024-09-04T21:54:52.740 INFO:teuthology.orchestra.run.smithi112.stdout: "read_tier": -1, 2024-09-04T21:54:52.740 INFO:teuthology.orchestra.run.smithi112.stdout: "write_tier": -1, 2024-09-04T21:54:52.740 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_mode": "none", 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_bytes": 0, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_objects": 0, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_flush_age": 0, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_evict_age": 0, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profile": "", 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_params": { 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "none" 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.741 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_period": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_count": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "use_gmt_hitset": true, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "min_read_recency_for_promote": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "min_write_recency_for_promote": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_search_last_n": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "grade_table": [], 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "stripe_width": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "expected_num_objects": 0, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "fast_read": false, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "options": {}, 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "application_metadata": { 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd": {} 2024-09-04T21:54:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "read_balance": { 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "score_type": "Fair distribution", 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "score_acting": 2.190000057220459, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "score_stable": 2.190000057220459, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "optimal_score": 0.86000001430511475, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity_weighted": 0.875, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity": 1, 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-04T21:54:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "osds": [ 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 0, 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "d210c577-fc3d-48c0-9acb-67afdd5a485a", 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 0, 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 66, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6816", 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6817", 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6818", 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6819", 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6822", 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6823", 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6820", 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6821", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6817/3808224845", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6819/3808224845", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6823/3808224845", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6821/3808224845", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 1, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "3d1c8d55-e5ac-4787-a725-6895bb2bf3f9", 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 0, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 22, 2024-09-04T21:54:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6824", 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6825", 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6826", 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6827", 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6830", 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6831", 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6828", 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6829", 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6825/245081828", 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6827/245081828", 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6831/245081828", 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6829/245081828", 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 2, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "5a67b7d1-ef0a-4935-9af0-01389e12d43e", 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 14, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 67, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6800", 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6801", 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6802", 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6803", 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6806", 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6807", 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6804", 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6805", 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6801/1590148387", 2024-09-04T21:54:52.760 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6803/1590148387", 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6807/1590148387", 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6805/1590148387", 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 3, 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "0b3ec943-bdb7-47e3-bcca-b52070713e9c", 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:54:52.761 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 14, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 67, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6808", 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6809", 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6810", 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6811", 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6814", 2024-09-04T21:54:52.764 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6815", 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6812", 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.766 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6813", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6809/2461796488", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6811/2461796488", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6815/2461796488", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6813/2461796488", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 4, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "09964dd2-405d-483d-822d-1b8c74497413", 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 67, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.768 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6824", 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6825", 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6826", 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6827", 2024-09-04T21:54:52.770 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6830", 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6831", 2024-09-04T21:54:52.771 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6828", 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6829", 2024-09-04T21:54:52.772 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3713217432 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6825/3713217432", 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6827/3713217432", 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6831/3713217432", 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6829/3713217432", 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 5, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "87306896-7b79-487f-ab06-ff337bb72f37", 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 0, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 13, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 48, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.774 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6807", 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6809", 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.775 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6810", 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6811", 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.777 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.777 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.777 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.777 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6814", 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6815", 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.778 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6812", 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6813", 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6809/352715412", 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6811/352715412", 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6815/352715412", 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6813/352715412", 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.780 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 6, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "e7948c3b-e9b4-4a9b-99be-8364f09b8c1f", 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 67, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.781 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6800", 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6801", 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6802", 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.783 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6803", 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6806", 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6808", 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.785 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6804", 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6805", 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6801/3465983375", 2024-09-04T21:54:52.786 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6803/3465983375", 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6808/3465983375", 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6805/3465983375", 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 7, 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "c9fdaad2-f75b-4dcd-a170-3e346fda0e16", 2024-09-04T21:54:52.787 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 13, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 21, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 48, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 67, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 22, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.788 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6816", 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6817", 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.789 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6818", 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6819", 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.790 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6822", 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6823", 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6820", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6821", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6817/471611328", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6819/471611328", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6823/471611328", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6821/471611328", 2024-09-04T21:54:52.792 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_xinfo": [ 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 0, 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.793 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 65536, 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.725388+0000", 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 1, 2024-09-04T21:54:52.794 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 65536, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:35.420675+0000", 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 2, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.795 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.389632+0000", 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 3, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:54:52.796 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.611983+0000", 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 4, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.796105+0000", 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 5, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.383432+0000", 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 6, 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.798 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.731214+0000", 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 7, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:54:52.799 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.469040+0000", 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 22 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_upmap": [], 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_upmap_items": [], 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_upmap_primaries": [], 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_temp": [], 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_temp": [], 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "blocklist": {}, 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "range_blocklist": {}, 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profiles": { 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "default": { 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "crush-failure-domain": "osd", 2024-09-04T21:54:52.800 INFO:teuthology.orchestra.run.smithi112.stdout: "k": "2", 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "m": "1", 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "plugin": "jerasure", 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "technique": "reed_sol_van" 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps_queue": [], 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "new_removed_snaps": [], 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "new_purged_snaps": [], 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_node_flags": {}, 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "device_class_flags": {}, 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": { 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode_enabled": false, 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_bucket_count": 0, 2024-09-04T21:54:52.801 INFO:teuthology.orchestra.run.smithi112.stdout: "degraded_stretch_mode": 0, 2024-09-04T21:54:52.802 INFO:teuthology.orchestra.run.smithi112.stdout: "recovering_stretch_mode": 0, 2024-09-04T21:54:52.802 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode_bucket": 0 2024-09-04T21:54:52.802 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:52.802 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:52.804 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:52.805 INFO:tasks.thrashosds.thrasher:j is {'epoch': 69, 'fsid': '7c7b20bb-5983-49a4-8b2c-296a55ea8d95', 'created': '2024-09-04T21:53:25.631463+0000', 'modified': '2024-09-04T21:54:52.354699+0000', 'last_up_change': '2024-09-04T21:54:30.565194+0000', 'last_in_change': '2024-09-04T21:54:47.345059+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-04T21:53:36.459454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-04T21:53:43.849962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-04T21:53:54.481634+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'd210c577-fc3d-48c0-9acb-67afdd5a485a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6816', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6817', 'nonce': 3808224845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6818', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6819', 'nonce': 3808224845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6822', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6823', 'nonce': 3808224845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6820', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6821', 'nonce': 3808224845}]}, 'public_addr': '172.21.15.112:6817/3808224845', 'cluster_addr': '172.21.15.112:6819/3808224845', 'heartbeat_back_addr': '172.21.15.112:6823/3808224845', 'heartbeat_front_addr': '172.21.15.112:6821/3808224845', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3d1c8d55-e5ac-4787-a725-6895bb2bf3f9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 245081828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 245081828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6830', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6831', 'nonce': 245081828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6828', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6829', 'nonce': 245081828}]}, 'public_addr': '172.21.15.112:6825/245081828', 'cluster_addr': '172.21.15.112:6827/245081828', 'heartbeat_back_addr': '172.21.15.112:6831/245081828', 'heartbeat_front_addr': '172.21.15.112:6829/245081828', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5a67b7d1-ef0a-4935-9af0-01389e12d43e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6800', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6801', 'nonce': 1590148387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6802', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6803', 'nonce': 1590148387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6806', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6807', 'nonce': 1590148387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6804', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6805', 'nonce': 1590148387}]}, 'public_addr': '172.21.15.112:6801/1590148387', 'cluster_addr': '172.21.15.112:6803/1590148387', 'heartbeat_back_addr': '172.21.15.112:6807/1590148387', 'heartbeat_front_addr': '172.21.15.112:6805/1590148387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0b3ec943-bdb7-47e3-bcca-b52070713e9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6808', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6809', 'nonce': 2461796488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6810', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6811', 'nonce': 2461796488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6814', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6815', 'nonce': 2461796488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6812', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6813', 'nonce': 2461796488}]}, 'public_addr': '172.21.15.112:6809/2461796488', 'cluster_addr': '172.21.15.112:6811/2461796488', 'heartbeat_back_addr': '172.21.15.112:6815/2461796488', 'heartbeat_front_addr': '172.21.15.112:6813/2461796488', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '09964dd2-405d-483d-822d-1b8c74497413', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6824', 'nonce': 3713217432}, {'type': 'v1', 'addr': '172.21.15.157:6825', 'nonce': 3713217432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6826', 'nonce': 3713217432}, {'type': 'v1', 'addr': '172.21.15.157:6827', 'nonce': 3713217432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6830', 'nonce': 3713217432}, {'type': 'v1', 'addr': '172.21.15.157:6831', 'nonce': 3713217432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6828', 'nonce': 3713217432}, {'type': 'v1', 'addr': '172.21.15.157:6829', 'nonce': 3713217432}]}, 'public_addr': '172.21.15.157:6825/3713217432', 'cluster_addr': '172.21.15.157:6827/3713217432', 'heartbeat_back_addr': '172.21.15.157:6831/3713217432', 'heartbeat_front_addr': '172.21.15.157:6829/3713217432', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '87306896-7b79-487f-ab06-ff337bb72f37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6807', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6809', 'nonce': 352715412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6810', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6811', 'nonce': 352715412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6814', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6815', 'nonce': 352715412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6812', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6813', 'nonce': 352715412}]}, 'public_addr': '172.21.15.157:6809/352715412', 'cluster_addr': '172.21.15.157:6811/352715412', 'heartbeat_back_addr': '172.21.15.157:6815/352715412', 'heartbeat_front_addr': '172.21.15.157:6813/352715412', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7948c3b-e9b4-4a9b-99be-8364f09b8c1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6800', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6801', 'nonce': 3465983375}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6802', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6803', 'nonce': 3465983375}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6806', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6808', 'nonce': 3465983375}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6804', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6805', 'nonce': 3465983375}]}, 'public_addr': '172.21.15.157:6801/3465983375', 'cluster_addr': '172.21.15.157:6803/3465983375', 'heartbeat_back_addr': '172.21.15.157:6808/3465983375', 'heartbeat_front_addr': '172.21.15.157:6805/3465983375', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9fdaad2-f75b-4dcd-a170-3e346fda0e16', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 21, 'up_from': 48, 'up_thru': 67, 'down_at': 22, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6816', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6817', 'nonce': 471611328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6818', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6819', 'nonce': 471611328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6822', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6823', 'nonce': 471611328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6820', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6821', 'nonce': 471611328}]}, 'public_addr': '172.21.15.157:6817/471611328', 'cluster_addr': '172.21.15.157:6819/471611328', 'heartbeat_back_addr': '172.21.15.157:6823/471611328', 'heartbeat_front_addr': '172.21.15.157:6821/471611328', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.725388+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-04T21:53:35.420675+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.389632+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.611983+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.796105+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.383432+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.731214+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.469040+0000', 'dead_epoch': 22}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-04T21:54:52.805 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:54:52.805 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:52.801+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:52.904 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:52.901+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:52.973 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:52.974 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:52.987 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:53.005 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:53.005+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:53.105 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:53.105 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:53.106 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:53.105+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:53.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:54:53.156 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:54:53.157 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:54:53.175 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-04T21:54:52.444074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545040059,"num_objects":662,"num_object_clones":208,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":208,"num_read":68992,"num_read_kb":402621,"num_write":9602,"num_write_kb":189467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":33244578,"num_keys_recovered":70,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7430,"ondisk_log_size":7430,"up":82,"acting":76,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":245752,"kb_used_data":27128,"kb_used_omap":12,"kb_used_meta":218483,"kb_avail":565985288,"statfs":{"total":579820584960,"available":579568934912,"internally_reserved":0,"allocated":27779072,"data_stored":220903580,"data_compressed":10819364,"data_compressed_allocated":19234816,"data_compressed_original":216332043,"omap_allocated":12366,"internal_metadata":223727538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18472174,"num_objects":60,"num_object_clones":30,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":26,"num_read":4225,"num_read_kb":22285,"num_write":664,"num_write_kb":12345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":5839417,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000899"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"unknown","last_fresh":"2024-09-04T21:54:47.366982+0000","last_change":"2024-09-04T21:54:47.701498+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-04T21:54:47.366982+0000","last_undegraded":"2024-09-04T21:54:47.366982+0000","last_fullsized":"2024-09-04T21:54:47.366982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'166","reported_seq":1752,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.365127+0000","last_change":"2024-09-04T21:53:57.370393+0000","last_active":"2024-09-04T21:54:47.365127+0000","last_peered":"2024-09-04T21:54:47.365127+0000","last_clean":"2024-09-04T21:54:47.365127+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:54:47.365127+0000","last_undegraded":"2024-09-04T21:54:47.365127+0000","last_fullsized":"2024-09-04T21:54:47.365127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:25:11.465130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372790000000002,"stat_sum":{"num_bytes":15033233,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":1516,"num_read_kb":8131,"num_write":162,"num_write_kb":4330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'352","reported_seq":1188,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.367440+0000","last_change":"2024-09-04T21:54:42.681915+0000","last_active":"2024-09-04T21:54:48.367440+0000","last_peered":"2024-09-04T21:54:48.367440+0000","last_clean":"2024-09-04T21:54:48.367440+0000","last_became_active":"2024-09-04T21:54:42.681042+0000","last_became_peered":"2024-09-04T21:54:42.681042+0000","last_unstale":"2024-09-04T21:54:48.367440+0000","last_undegraded":"2024-09-04T21:54:48.367440+0000","last_fullsized":"2024-09-04T21:54:48.367440+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:51:48.615021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00153718,"stat_sum":{"num_bytes":32334288,"num_objects":22,"num_object_clones":8,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":6,"num_read":2029,"num_read_kb":5647,"num_write":538,"num_write_kb":6477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'193","reported_seq":1763,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.366912+0000","last_change":"2024-09-04T21:54:42.680392+0000","last_active":"2024-09-04T21:54:48.366912+0000","last_peered":"2024-09-04T21:54:48.366912+0000","last_clean":"2024-09-04T21:54:48.366912+0000","last_became_active":"2024-09-04T21:54:42.679589+0000","last_became_peered":"2024-09-04T21:54:42.679589+0000","last_unstale":"2024-09-04T21:54:48.366912+0000","last_undegraded":"2024-09-04T21:54:48.366912+0000","last_fullsized":"2024-09-04T21:54:48.366912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:58:48.826628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349371,"stat_sum":{"num_bytes":18683528,"num_objects":22,"num_object_clones":7,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":10,"num_read":1526,"num_read_kb":19158,"num_write":184,"num_write_kb":3814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"62'138","reported_seq":887,"reported_epoch":65,"state":"remapped+peering","last_fresh":"2024-09-04T21:54:48.367460+0000","last_change":"2024-09-04T21:54:48.367460+0000","last_active":"2024-09-04T21:54:47.356692+0000","last_peered":"2024-09-04T21:54:45.477085+0000","last_clean":"2024-09-04T21:54:45.477085+0000","last_became_active":"2024-09-04T21:53:55.356638+0000","last_became_peered":"2024-09-04T21:53:55.356638+0000","last_unstale":"2024-09-04T21:54:48.367460+0000","last_undegraded":"2024-09-04T21:54:48.367460+0000","last_fullsized":"2024-09-04T21:54:48.367460+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295954,"stat_sum":{"num_bytes":5216048,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":5,"num_read":688,"num_read_kb":5071,"num_write":136,"num_write_kb":5432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"63'231","reported_seq":3997,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.367929+0000","last_change":"2024-09-04T21:54:13.554263+0000","last_active":"2024-09-04T21:54:48.367929+0000","last_peered":"2024-09-04T21:54:48.367929+0000","last_clean":"2024-09-04T21:54:48.367929+0000","last_became_active":"2024-09-04T21:54:13.481244+0000","last_became_peered":"2024-09-04T21:54:13.481244+0000","last_unstale":"2024-09-04T21:54:48.367929+0000","last_undegraded":"2024-09-04T21:54:48.367929+0000","last_fullsized":"2024-09-04T21:54:48.367929+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:11:02.215455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001385184,"stat_sum":{"num_bytes":28476090,"num_objects":29,"num_object_clones":9,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":9,"num_read":3837,"num_read_kb":35644,"num_write":308,"num_write_kb":7040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":930020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"63'799","reported_seq":1354,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.363457+0000","last_change":"2024-09-04T21:54:13.510809+0000","last_active":"2024-09-04T21:54:47.363457+0000","last_peered":"2024-09-04T21:54:47.363457+0000","last_clean":"2024-09-04T21:54:47.363457+0000","last_became_active":"2024-09-04T21:54:13.478356+0000","last_became_peered":"2024-09-04T21:54:13.478356+0000","last_unstale":"2024-09-04T21:54:47.363457+0000","last_undegraded":"2024-09-04T21:54:47.363457+0000","last_fullsized":"2024-09-04T21:54:47.363457+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:28:01.616615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032359149999999998,"stat_sum":{"num_bytes":10081381,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":6,"num_read":3944,"num_read_kb":13786,"num_write":1366,"num_write_kb":5009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"62'225","reported_seq":2913,"reported_epoch":63,"state":"peering","last_fresh":"2024-09-04T21:54:46.375976+0000","last_change":"2024-09-04T21:54:47.363628+0000","last_active":"2024-09-04T21:54:47.363628+0000","last_peered":"2024-09-04T21:54:46.375976+0000","last_clean":"2024-09-04T21:54:46.375976+0000","last_became_active":"2024-09-04T21:53:55.349441+0000","last_became_peered":"2024-09-04T21:53:55.349441+0000","last_unstale":"2024-09-04T21:54:46.375976+0000","last_undegraded":"2024-09-04T21:54:46.375976+0000","last_fullsized":"2024-09-04T21:54:46.375976+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001549499,"stat_sum":{"num_bytes":21370105,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":5,"num_read":3969,"num_read_kb":23996,"num_write":250,"num_write_kb":6992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"unknown","last_fresh":"2024-09-04T21:54:47.365451+0000","last_change":"2024-09-04T21:54:47.703193+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-04T21:54:47.365451+0000","last_undegraded":"2024-09-04T21:54:47.365451+0000","last_fullsized":"2024-09-04T21:54:47.365451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"63'91","reported_seq":946,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362957+0000","last_change":"2024-09-04T21:54:42.680486+0000","last_active":"2024-09-04T21:54:49.362957+0000","last_peered":"2024-09-04T21:54:49.362957+0000","last_clean":"2024-09-04T21:54:49.362957+0000","last_became_active":"2024-09-04T21:54:42.679389+0000","last_became_peered":"2024-09-04T21:54:42.679389+0000","last_unstale":"2024-09-04T21:54:49.362957+0000","last_undegraded":"2024-09-04T21:54:49.362957+0000","last_fullsized":"2024-09-04T21:54:49.362957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T21:54:44.807737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086423499999999998,"stat_sum":{"num_bytes":12429312,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":712,"num_read_kb":12968,"num_write":94,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"63'159","reported_seq":1605,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.363037+0000","last_change":"2024-09-04T21:54:31.683707+0000","last_active":"2024-09-04T21:54:49.363037+0000","last_peered":"2024-09-04T21:54:49.363037+0000","last_clean":"2024-09-04T21:54:49.363037+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:54:49.363037+0000","last_undegraded":"2024-09-04T21:54:49.363037+0000","last_fullsized":"2024-09-04T21:54:49.363037+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:54:58.451603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078592199999999999,"stat_sum":{"num_bytes":17984373,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":16,"num_read":1830,"num_read_kb":17856,"num_write":160,"num_write_kb":14096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"63'124","reported_seq":740,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.367145+0000","last_change":"2024-09-04T21:54:31.673826+0000","last_active":"2024-09-04T21:54:48.367145+0000","last_peered":"2024-09-04T21:54:48.367145+0000","last_clean":"2024-09-04T21:54:48.367145+0000","last_became_active":"2024-09-04T21:54:31.582403+0000","last_became_peered":"2024-09-04T21:54:31.582403+0000","last_unstale":"2024-09-04T21:54:48.367145+0000","last_undegraded":"2024-09-04T21:54:48.367145+0000","last_fullsized":"2024-09-04T21:54:48.367145+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:26:39.717637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001035425,"stat_sum":{"num_bytes":4592417,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":6,"num_read":559,"num_read_kb":2068,"num_write":149,"num_write_kb":3215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":529142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.366754+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:54:48.366754+0000","last_peered":"2024-09-04T21:54:48.366754+0000","last_clean":"2024-09-04T21:54:48.366754+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:54:48.366754+0000","last_undegraded":"2024-09-04T21:54:48.366754+0000","last_fullsized":"2024-09-04T21:54:48.366754+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"63'154","reported_seq":646,"reported_epoch":65,"state":"remapped+peering","last_fresh":"2024-09-04T21:54:48.369339+0000","last_change":"2024-09-04T21:54:48.369339+0000","last_active":"2024-09-04T21:54:47.356806+0000","last_peered":"2024-09-04T21:54:46.611476+0000","last_clean":"2024-09-04T21:54:46.611476+0000","last_became_active":"2024-09-04T21:54:13.481286+0000","last_became_peered":"2024-09-04T21:54:13.481286+0000","last_unstale":"2024-09-04T21:54:48.369339+0000","last_undegraded":"2024-09-04T21:54:48.369339+0000","last_fullsized":"2024-09-04T21:54:48.369339+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001390173,"stat_sum":{"num_bytes":4021866,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":518,"num_read_kb":1099,"num_write":156,"num_write_kb":2115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":90858,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":70,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.366766+0000","last_change":"2024-09-04T21:54:13.481230+0000","last_active":"2024-09-04T21:54:48.366766+0000","last_peered":"2024-09-04T21:54:48.366766+0000","last_clean":"2024-09-04T21:54:48.366766+0000","last_became_active":"2024-09-04T21:54:13.479390+0000","last_became_peered":"2024-09-04T21:54:13.479390+0000","last_unstale":"2024-09-04T21:54:48.366766+0000","last_undegraded":"2024-09-04T21:54:48.366766+0000","last_fullsized":"2024-09-04T21:54:48.366766+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:05.478515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"63'148","reported_seq":2512,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.365048+0000","last_change":"2024-09-04T21:54:31.773042+0000","last_active":"2024-09-04T21:54:47.365048+0000","last_peered":"2024-09-04T21:54:47.365048+0000","last_clean":"2024-09-04T21:54:47.365048+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:54:47.365048+0000","last_undegraded":"2024-09-04T21:54:47.365048+0000","last_fullsized":"2024-09-04T21:54:47.365048+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:50:06.646577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015346380000000001,"stat_sum":{"num_bytes":9240138,"num_objects":27,"num_object_clones":10,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":7,"num_read":2808,"num_read_kb":18062,"num_write":191,"num_write_kb":8714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.363293+0000","last_change":"2024-09-04T21:54:48.374647+0000","last_active":"2024-09-04T21:54:49.363293+0000","last_peered":"2024-09-04T21:54:49.363293+0000","last_clean":"2024-09-04T21:54:49.363293+0000","last_became_active":"2024-09-04T21:54:48.374288+0000","last_became_peered":"2024-09-04T21:54:48.374288+0000","last_unstale":"2024-09-04T21:54:49.363293+0000","last_undegraded":"2024-09-04T21:54:49.363293+0000","last_fullsized":"2024-09-04T21:54:49.363293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:14:06.038769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"62'153","reported_seq":1146,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.367962+0000","last_change":"2024-09-04T21:54:13.539266+0000","last_active":"2024-09-04T21:54:47.367962+0000","last_peered":"2024-09-04T21:54:47.367962+0000","last_clean":"2024-09-04T21:54:47.367962+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:54:47.367962+0000","last_undegraded":"2024-09-04T21:54:47.367962+0000","last_fullsized":"2024-09-04T21:54:47.367962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:16:29.506926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091729799999999998,"stat_sum":{"num_bytes":5747236,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1498,"num_read_kb":5438,"num_write":155,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362885+0000","last_change":"2024-09-04T21:54:48.372640+0000","last_active":"2024-09-04T21:54:49.362885+0000","last_peered":"2024-09-04T21:54:49.362885+0000","last_clean":"2024-09-04T21:54:49.362885+0000","last_became_active":"2024-09-04T21:54:48.372106+0000","last_became_peered":"2024-09-04T21:54:48.372106+0000","last_unstale":"2024-09-04T21:54:49.362885+0000","last_undegraded":"2024-09-04T21:54:49.362885+0000","last_fullsized":"2024-09-04T21:54:49.362885+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:18:59.210273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"63'297","reported_seq":2034,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.362520+0000","last_change":"2024-09-04T21:54:42.680694+0000","last_active":"2024-09-04T21:54:47.362520+0000","last_peered":"2024-09-04T21:54:47.362520+0000","last_clean":"2024-09-04T21:54:47.362520+0000","last_became_active":"2024-09-04T21:54:42.679683+0000","last_became_peered":"2024-09-04T21:54:42.679683+0000","last_unstale":"2024-09-04T21:54:47.362520+0000","last_undegraded":"2024-09-04T21:54:47.362520+0000","last_fullsized":"2024-09-04T21:54:47.362520+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:30:56.753910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012888800000000001,"stat_sum":{"num_bytes":24702247,"num_objects":22,"num_object_clones":7,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":5,"num_read":2793,"num_read_kb":16700,"num_write":387,"num_write_kb":8994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"63'172","reported_seq":2649,"reported_epoch":64,"state":"remapped+peering","last_fresh":"2024-09-04T21:54:47.357587+0000","last_change":"2024-09-04T21:54:48.368352+0000","last_active":"2024-09-04T21:54:47.357587+0000","last_peered":"2024-09-04T21:54:46.536007+0000","last_clean":"2024-09-04T21:54:46.536007+0000","last_became_active":"2024-09-04T21:53:55.349651+0000","last_became_peered":"2024-09-04T21:53:55.349651+0000","last_unstale":"2024-09-04T21:54:47.357587+0000","last_undegraded":"2024-09-04T21:54:47.357587+0000","last_fullsized":"2024-09-04T21:54:47.357587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050067700000000004,"stat_sum":{"num_bytes":12898023,"num_objects":22,"num_object_clones":7,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":7,"num_read":2467,"num_read_kb":14444,"num_write":194,"num_write_kb":11042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":70,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.366614+0000","last_change":"2024-09-04T21:54:42.675259+0000","last_active":"2024-09-04T21:54:48.366614+0000","last_peered":"2024-09-04T21:54:48.366614+0000","last_clean":"2024-09-04T21:54:48.366614+0000","last_became_active":"2024-09-04T21:54:42.674803+0000","last_became_peered":"2024-09-04T21:54:42.674803+0000","last_unstale":"2024-09-04T21:54:48.366614+0000","last_undegraded":"2024-09-04T21:54:48.366614+0000","last_fullsized":"2024-09-04T21:54:48.366614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:09:21.260817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"63'223","reported_seq":2028,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.367598+0000","last_change":"2024-09-04T21:54:16.342878+0000","last_active":"2024-09-04T21:54:47.367598+0000","last_peered":"2024-09-04T21:54:47.367598+0000","last_clean":"2024-09-04T21:54:47.367598+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:54:47.367598+0000","last_undegraded":"2024-09-04T21:54:47.367598+0000","last_fullsized":"2024-09-04T21:54:47.367598+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:35:30.037310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010876340000000001,"stat_sum":{"num_bytes":13736313,"num_objects":32,"num_object_clones":9,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":13,"num_read":2726,"num_read_kb":8601,"num_write":207,"num_write_kb":4358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.367840+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:54:47.367840+0000","last_peered":"2024-09-04T21:54:47.367840+0000","last_clean":"2024-09-04T21:54:47.367840+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:54:47.367840+0000","last_undegraded":"2024-09-04T21:54:47.367840+0000","last_fullsized":"2024-09-04T21:54:47.367840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"63'308","reported_seq":2492,"reported_epoch":65,"state":"remapped+peering","last_fresh":"2024-09-04T21:54:48.367380+0000","last_change":"2024-09-04T21:54:48.367380+0000","last_active":"2024-09-04T21:54:47.357154+0000","last_peered":"2024-09-04T21:54:46.461459+0000","last_clean":"2024-09-04T21:54:46.461459+0000","last_became_active":"2024-09-04T21:53:55.354821+0000","last_became_peered":"2024-09-04T21:53:55.354821+0000","last_unstale":"2024-09-04T21:54:48.367380+0000","last_undegraded":"2024-09-04T21:54:48.367380+0000","last_fullsized":"2024-09-04T21:54:48.367380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396415,"stat_sum":{"num_bytes":41564918,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":3101,"num_read_kb":24346,"num_write":461,"num_write_kb":5123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":69,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362650+0000","last_change":"2024-09-04T21:54:48.373076+0000","last_active":"2024-09-04T21:54:49.362650+0000","last_peered":"2024-09-04T21:54:49.362650+0000","last_clean":"2024-09-04T21:54:49.362650+0000","last_became_active":"2024-09-04T21:54:48.372613+0000","last_became_peered":"2024-09-04T21:54:48.372613+0000","last_unstale":"2024-09-04T21:54:49.362650+0000","last_undegraded":"2024-09-04T21:54:49.362650+0000","last_fullsized":"2024-09-04T21:54:49.362650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:48:13.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"63'236","reported_seq":2573,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.362932+0000","last_change":"2024-09-04T21:53:57.369713+0000","last_active":"2024-09-04T21:54:47.362932+0000","last_peered":"2024-09-04T21:54:47.362932+0000","last_clean":"2024-09-04T21:54:47.362932+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:54:47.362932+0000","last_undegraded":"2024-09-04T21:54:47.362932+0000","last_fullsized":"2024-09-04T21:54:47.362932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:33:10.545336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026956770000000001,"stat_sum":{"num_bytes":26737179,"num_objects":29,"num_object_clones":7,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":5,"num_read":3296,"num_read_kb":12091,"num_write":236,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":28,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.368015+0000","last_change":"2024-09-04T21:54:42.676579+0000","last_active":"2024-09-04T21:54:47.368015+0000","last_peered":"2024-09-04T21:54:47.368015+0000","last_clean":"2024-09-04T21:54:47.368015+0000","last_became_active":"2024-09-04T21:54:42.676301+0000","last_became_peered":"2024-09-04T21:54:42.676301+0000","last_unstale":"2024-09-04T21:54:47.368015+0000","last_undegraded":"2024-09-04T21:54:47.368015+0000","last_fullsized":"2024-09-04T21:54:47.368015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:42:21.618311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":159,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362449+0000","last_change":"2024-09-04T21:54:48.383973+0000","last_active":"2024-09-04T21:54:49.362449+0000","last_peered":"2024-09-04T21:54:49.362449+0000","last_clean":"2024-09-04T21:54:49.362449+0000","last_became_active":"2024-09-04T21:54:48.374571+0000","last_became_peered":"2024-09-04T21:54:48.374571+0000","last_unstale":"2024-09-04T21:54:49.362449+0000","last_undegraded":"2024-09-04T21:54:49.362449+0000","last_fullsized":"2024-09-04T21:54:49.362449+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:54:44.340649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"63'399","reported_seq":2615,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.363940+0000","last_change":"2024-09-04T21:54:34.607151+0000","last_active":"2024-09-04T21:54:47.363940+0000","last_peered":"2024-09-04T21:54:47.363940+0000","last_clean":"2024-09-04T21:54:47.363940+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:54:47.363940+0000","last_undegraded":"2024-09-04T21:54:47.363940+0000","last_fullsized":"2024-09-04T21:54:47.363940+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:11:49.409640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001397155,"stat_sum":{"num_bytes":20223384,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":4,"num_read":3795,"num_read_kb":13039,"num_write":540,"num_write_kb":5974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"63'162","reported_seq":1376,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.367463+0000","last_change":"2024-09-04T21:53:57.346105+0000","last_active":"2024-09-04T21:54:48.367463+0000","last_peered":"2024-09-04T21:54:48.367463+0000","last_clean":"2024-09-04T21:54:48.367463+0000","last_became_active":"2024-09-04T21:53:55.345842+0000","last_became_peered":"2024-09-04T21:53:55.345842+0000","last_unstale":"2024-09-04T21:54:48.367463+0000","last_undegraded":"2024-09-04T21:54:48.367463+0000","last_fullsized":"2024-09-04T21:54:48.367463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:55:29.806692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094585800000000005,"stat_sum":{"num_bytes":13389978,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":1129,"num_read_kb":12075,"num_write":153,"num_write_kb":4833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"63'419","reported_seq":1831,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-04T21:54:48.366722+0000","last_change":"2024-09-04T21:53:57.339901+0000","last_active":"2024-09-04T21:54:48.366722+0000","last_peered":"2024-09-04T21:54:48.366722+0000","last_clean":"2024-09-04T21:54:48.366722+0000","last_became_active":"2024-09-04T21:53:55.355393+0000","last_became_peered":"2024-09-04T21:53:55.355393+0000","last_unstale":"2024-09-04T21:54:48.366722+0000","last_undegraded":"2024-09-04T21:54:48.366722+0000","last_fullsized":"2024-09-04T21:54:48.366722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:26:08.073794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050858999999999995,"stat_sum":{"num_bytes":51964984,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":3157,"num_read_kb":10241,"num_write":583,"num_write_kb":5190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"63'206","reported_seq":942,"reported_epoch":63,"state":"peering","last_fresh":"2024-09-04T21:54:46.463856+0000","last_change":"2024-09-04T21:54:47.364628+0000","last_active":"2024-09-04T21:54:47.364628+0000","last_peered":"2024-09-04T21:54:46.463856+0000","last_clean":"2024-09-04T21:54:46.463856+0000","last_became_active":"2024-09-04T21:53:55.349356+0000","last_became_peered":"2024-09-04T21:53:55.349356+0000","last_unstale":"2024-09-04T21:54:46.463856+0000","last_undegraded":"2024-09-04T21:54:46.463856+0000","last_fullsized":"2024-09-04T21:54:46.463856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024267529999999998,"stat_sum":{"num_bytes":3571780,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":4,"num_read":1956,"num_read_kb":4518,"num_write":273,"num_write_kb":4141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"63'144","reported_seq":1690,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-04T21:54:47.362697+0000","last_change":"2024-09-04T21:54:42.680365+0000","last_active":"2024-09-04T21:54:47.362697+0000","last_peered":"2024-09-04T21:54:47.362697+0000","last_clean":"2024-09-04T21:54:47.362697+0000","last_became_active":"2024-09-04T21:54:42.679512+0000","last_became_peered":"2024-09-04T21:54:42.679512+0000","last_unstale":"2024-09-04T21:54:47.362697+0000","last_undegraded":"2024-09-04T21:54:47.362697+0000","last_fullsized":"2024-09-04T21:54:47.362697+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:25:08.423039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361924,"stat_sum":{"num_bytes":34590797,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":8,"num_read":1477,"num_read_kb":9313,"num_write":150,"num_write_kb":4834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"63'124","reported_seq":406,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362686+0000","last_change":"2024-09-04T21:54:31.623898+0000","last_active":"2024-09-04T21:54:49.362686+0000","last_peered":"2024-09-04T21:54:49.362686+0000","last_clean":"2024-09-04T21:54:49.362686+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:54:49.362686+0000","last_undegraded":"2024-09-04T21:54:49.362686+0000","last_fullsized":"2024-09-04T21:54:49.362686+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:32:14.306727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091741999999999995,"stat_sum":{"num_bytes":6094765,"num_objects":20,"num_object_clones":8,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":9,"num_read":752,"num_read_kb":6108,"num_write":122,"num_write_kb":3164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"63'132","reported_seq":504,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.363316+0000","last_change":"2024-09-04T21:54:31.719167+0000","last_active":"2024-09-04T21:54:49.363316+0000","last_peered":"2024-09-04T21:54:49.363316+0000","last_clean":"2024-09-04T21:54:49.363316+0000","last_became_active":"2024-09-04T21:54:31.584135+0000","last_became_peered":"2024-09-04T21:54:31.584135+0000","last_unstale":"2024-09-04T21:54:49.363316+0000","last_undegraded":"2024-09-04T21:54:49.363316+0000","last_fullsized":"2024-09-04T21:54:49.363316+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:23:43.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066328299999999997,"stat_sum":{"num_bytes":13077279,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":9,"num_read":1478,"num_read_kb":9617,"num_write":177,"num_write_kb":4461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":894295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'216","reported_seq":820,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.363048+0000","last_change":"2024-09-04T21:54:34.612521+0000","last_active":"2024-09-04T21:54:49.363048+0000","last_peered":"2024-09-04T21:54:49.363048+0000","last_clean":"2024-09-04T21:54:49.363048+0000","last_became_active":"2024-09-04T21:54:34.608209+0000","last_became_peered":"2024-09-04T21:54:34.608209+0000","last_unstale":"2024-09-04T21:54:49.363048+0000","last_undegraded":"2024-09-04T21:54:49.363048+0000","last_fullsized":"2024-09-04T21:54:49.363048+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:41:39.049516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025152799999999999,"stat_sum":{"num_bytes":7279862,"num_objects":15,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":5,"num_read":1040,"num_read_kb":1865,"num_write":247,"num_write_kb":2887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":717304,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'425","reported_seq":2136,"reported_epoch":64,"state":"remapped+peering","last_fresh":"2024-09-04T21:54:47.360632+0000","last_change":"2024-09-04T21:54:48.368150+0000","last_active":"2024-09-04T21:54:47.360632+0000","last_peered":"2024-09-04T21:54:46.534724+0000","last_clean":"2024-09-04T21:54:46.534724+0000","last_became_active":"2024-09-04T21:53:55.352363+0000","last_became_peered":"2024-09-04T21:53:55.352363+0000","last_unstale":"2024-09-04T21:54:47.360632+0000","last_undegraded":"2024-09-04T21:54:47.360632+0000","last_fullsized":"2024-09-04T21:54:47.360632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000478915,"stat_sum":{"num_bytes":6798626,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":11,"num_read":3776,"num_read_kb":10072,"num_write":567,"num_write_kb":7290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"62'398","reported_seq":4928,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362614+0000","last_change":"2024-09-04T21:54:34.612100+0000","last_active":"2024-09-04T21:54:49.362614+0000","last_peered":"2024-09-04T21:54:49.362614+0000","last_clean":"2024-09-04T21:54:49.362614+0000","last_became_active":"2024-09-04T21:54:34.607984+0000","last_became_peered":"2024-09-04T21:54:34.607984+0000","last_unstale":"2024-09-04T21:54:49.362614+0000","last_undegraded":"2024-09-04T21:54:49.362614+0000","last_fullsized":"2024-09-04T21:54:49.362614+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:13:21.150172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023455609999999999,"stat_sum":{"num_bytes":29871080,"num_objects":26,"num_object_clones":7,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":6131,"num_read_kb":41669,"num_write":551,"num_write_kb":24482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":6482454,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"66'240","reported_seq":2947,"reported_epoch":66,"state":"active+undersized+remapped","last_fresh":"2024-09-04T21:54:49.590801+0000","last_change":"2024-09-04T21:54:49.546467+0000","last_active":"2024-09-04T21:54:49.590801+0000","last_peered":"2024-09-04T21:54:49.590801+0000","last_clean":"2024-09-04T21:54:46.345784+0000","last_became_active":"2024-09-04T21:54:49.387400+0000","last_became_peered":"2024-09-04T21:54:49.387400+0000","last_unstale":"2024-09-04T21:54:49.590801+0000","last_undegraded":"2024-09-04T21:54:49.590801+0000","last_fullsized":"2024-09-04T21:54:49.362743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010422490000000001,"stat_sum":{"num_bytes":36534858,"num_objects":35,"num_object_clones":14,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":10,"num_read":2844,"num_read_kb":27340,"num_write":305,"num_write_kb":7738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":7361343,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"7","objects":35}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"63'321","reported_seq":1015,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-04T21:54:49.362623+0000","last_change":"2024-09-04T21:54:34.609339+0000","last_active":"2024-09-04T21:54:49.362623+0000","last_peered":"2024-09-04T21:54:49.362623+0000","last_clean":"2024-09-04T21:54:49.362623+0000","last_became_active":"2024-09-04T21:54:34.607781+0000","last_became_peered":"2024-09-04T21:54:34.607781+0000","last_unstale":"2024-09-04T21:54:49.362623+0000","last_undegraded":"2024-09-04T21:54:49.362623+0000","last_fullsized":"2024-09-04T21:54:49.362623+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:48:53.442681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091209699999999995,"stat_sum":{"num_bytes":16203584,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":6,"num_read":1574,"num_read_kb":11603,"num_write":497,"num_write_kb":4619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1583654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":544449672,"num_objects":658,"num_object_clones":208,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":208,"num_read":68926,"num_read_kb":402566,"num_write":9499,"num_write_kb":187837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":32654210,"num_keys_recovered":70,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27594752,"data_stored":255066825,"data_compressed":13064262,"data_compressed_allocated":22745088,"data_compressed_original":252003382,"omap_allocated":0,"internal_metadata":0},"log_size":7355,"ondisk_log_size":7355,"up":64,"acting":58,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":18583,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509458,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41752,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330088,"statfs":{"total":96636764160,"available":96594010112,"internally_reserved":0,"allocated":2297856,"data_stored":17146239,"data_compressed":681223,"data_compressed_allocated":1290240,"data_compressed_original":16669902,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55692,"kb_used_data":6456,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94316148,"statfs":{"total":96636764160,"available":96579735552,"internally_reserved":0,"allocated":6610944,"data_stored":51025503,"data_compressed":2881737,"data_compressed_allocated":4739072,"data_compressed_original":49997959,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542161,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44792,"kb_used_data":5284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94327048,"statfs":{"total":96636764160,"available":96590897152,"internally_reserved":0,"allocated":5410816,"data_stored":47327497,"data_compressed":2127421,"data_compressed_allocated":3903488,"data_compressed_original":46459959,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33452,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338388,"statfs":{"total":96636764160,"available":96602509312,"internally_reserved":0,"allocated":2318336,"data_stored":16097377,"data_compressed":746244,"data_compressed_allocated":1323008,"data_compressed_original":15601619,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38388,"kb_used_data":6240,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94333452,"statfs":{"total":96636764160,"available":96597454848,"internally_reserved":0,"allocated":6389760,"data_stored":52630739,"data_compressed":2497768,"data_compressed_allocated":4730880,"data_compressed_original":51774705,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":4751360,"data_stored":36676225,"data_compressed":1884971,"data_compressed_allocated":3248128,"data_compressed_original":35827899,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":35643695,"data_compressed":2027336,"data_compressed_allocated":3260416,"data_compressed_original":35162080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1581056,"data_stored":15763655,"data_compressed":746244,"data_compressed_allocated":1323008,"data_compressed_original":15601619,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4673536,"data_stored":46993775,"data_compressed":2127421,"data_compressed_allocated":3903488,"data_compressed_original":46459959,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5873664,"data_stored":50691781,"data_compressed":2881737,"data_compressed_allocated":4739072,"data_compressed_original":49997959,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4030464,"data_stored":36376109,"data_compressed":1923465,"data_compressed_allocated":3284992,"data_compressed_original":35864763,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1503232,"data_stored":16215769,"data_compressed":674488,"data_compressed_allocated":1253376,"data_compressed_original":16080078,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5570560,"data_stored":51694185,"data_compressed":2491844,"data_compressed_allocated":4694016,"data_compressed_original":51201265,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:54:53.177 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [3, 4, 1, 6] 2024-09-04T21:54:53.177 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '3', '4', '1', '6'] 2024-09-04T21:54:53.177 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 3 4 1 6 2024-09-04T21:54:53.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:53.221 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:53.221+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:53.236 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:53.236 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:53.250 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:54:53.307 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:53.305+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:53.360 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:53.361 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:53.375 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:53.375 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:53.381 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:53.381 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:53.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:53.398 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:53.399 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:53.407 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:53.405+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:53.416 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:53.416 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:53.438 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:53.439 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:53.459 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:53.460 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:53.473 INFO:teuthology.orchestra.run.smithi112.stdout:leaving image image_client.0-clone19 intact 2024-09-04T21:54:53.485 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:53.485 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:53.506 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:53.506 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:53.507 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:53.505+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:53.517 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:53.517 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:53.517 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:53.517 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:53.535 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:53.629 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:53.629+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:53.668 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-09-04T21:54:53.698 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:53.698 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:53.698 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:53.698 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:53.698 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:53.710 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:53.705+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:53.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:53.810 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:53.805+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:53.846 INFO:teuthology.orchestra.run.smithi112.stdout:3996 trunc from 0xc44a1a9 to 0x5f709a2 2024-09-04T21:54:53.874 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:53.874 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:53.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:53.874 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:53.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.874 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198631722000002, 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:53.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:53.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:53.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:53.879 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198604936999999, 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:53.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:53.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198576004000003, 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:53.885 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:54.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:54.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198552356999997, 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.005 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:54.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:54.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198532575999998, 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:54.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198513181000003, 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.014 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:54.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198493908000003, 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:54.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:54.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198474685999997, 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.023 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:54.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.027 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198455848999998, 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.028 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:54.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198436446000002, 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.033 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:54.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:54.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.037 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198415390000001, 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:54.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198383212000003, 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:54.042 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.046 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198356990999997, 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.051 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198328781999997, 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.052 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:54.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198308742000002, 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.056 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:54.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198289359999997, 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:54.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.065 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198265771999999, 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:54.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198245454999999, 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.070 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:54.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198225903000001, 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.075 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:54.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.198206143999997, 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:54.079 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:54.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.083 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.084 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.084 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:54.085 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:3997 write 0xe465ee6 thru 0xe46f372 (0x948d bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:3998 punch from 0xc7fd70a to 0xc80d073, (0xf969 bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:3999 read 0xcc845cf thru 0xcc8b252 (0x6c84 bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4000 writesame 0x516ac01 thru 0x517435d data_size 0x1(0x975d bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4002 punch from 0x91367ab to 0x91432e0, (0xcb35 bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4003 punch from 0x107c3d7 to 0x1088f61, (0xcb8a bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4004 write 0xb437322 thru 0xb44023a (0x8f19 bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4005 trunc from 0xe46f373 to 0x4186e11 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4006 trunc from 0x4186e11 to 0xdf1dcf1 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4007 write 0x48a6e78 thru 0x48b313e (0xc2c7 bytes) 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.087 INFO:teuthology.orchestra.run.smithi112.stdout:4009 read 0x64900e1 thru 0x649de73 (0xdd93 bytes) 2024-09-04T21:54:54.088 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:54.085+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:54.088 INFO:teuthology.orchestra.run.smithi112.stdout:4010 trunc from 0xdf1dcf1 to 0x4dfc6a 2024-09-04T21:54:54.116 INFO:teuthology.orchestra.run.smithi112.stdout:4011 read 0x1f087e thru 0x1f3510 (0x2c93 bytes) 2024-09-04T21:54:54.152 INFO:teuthology.orchestra.run.smithi112.stdout:4012 write 0xe6902fd thru 0xe69e09c (0xdda0 bytes) 2024-09-04T21:54:54.158 INFO:teuthology.orchestra.run.smithi112.stdout:4013 flatten 2024-09-04T21:54:54.186 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:54.185+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:54.196 INFO:teuthology.orchestra.run.smithi112.stdout:4015 write 0x6a8d350 thru 0x6a9b1ec (0xde9d bytes) 2024-09-04T21:54:54.209 INFO:teuthology.orchestra.run.smithi112.stdout:4016 write 0x5bcb9b7 thru 0x5bdb8ef (0xff39 bytes) 2024-09-04T21:54:54.217 INFO:teuthology.orchestra.run.smithi112.stdout:4017 writesame 0x14b6e6a thru 0x14c132b data_size 0x1(0xa4c2 bytes) 2024-09-04T21:54:54.224 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:54.224 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:54.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:54.274 INFO:teuthology.orchestra.run.smithi112.stdout:4018 punch from 0xa86cd2e to 0xa875637, (0x8909 bytes) 2024-09-04T21:54:54.274 INFO:teuthology.orchestra.run.smithi112.stdout:4019 write 0x1bdaf19 thru 0x1be5410 (0xa4f8 bytes) 2024-09-04T21:54:54.283 INFO:teuthology.orchestra.run.smithi112.stdout:4020 read 0xe66dfe9 thru 0xe6713f5 (0x340d bytes) 2024-09-04T21:54:54.287 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:54.285+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:54.295 INFO:teuthology.orchestra.run.smithi112.stdout:4021 trunc from 0xe69e09d to 0x6214521 2024-09-04T21:54:54.321 INFO:teuthology.orchestra.run.smithi112.stdout:4022 write 0x6eb66be thru 0x6ec256b (0xbeae bytes) 2024-09-04T21:54:54.329 INFO:teuthology.orchestra.run.smithi112.stdout:4024 punch from 0x60a9524 to 0x60afb37, (0x6613 bytes) 2024-09-04T21:54:54.329 INFO:teuthology.orchestra.run.smithi112.stdout:4026 write 0x1b88c44 thru 0x1b984a5 (0xf862 bytes) 2024-09-04T21:54:54.352 INFO:teuthology.orchestra.run.smithi112.stdout:4027 write 0xc5c2b9f thru 0xc5c722b (0x468d bytes) 2024-09-04T21:54:54.359 INFO:teuthology.orchestra.run.smithi112.stdout:4028 read 0xc538f8d thru 0xc53c2c9 (0x333d bytes) 2024-09-04T21:54:54.360 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.360 INFO:teuthology.orchestra.run.smithi112.stdout:4030 read 0x7457cc7 thru 0x74583d8 (0x712 bytes) 2024-09-04T21:54:54.360 INFO:teuthology.orchestra.run.smithi112.stdout:4031 writesame 0x4278bc8 thru 0x427a5e6 data_size 0x3(0x1a1f bytes) 2024-09-04T21:54:54.368 INFO:teuthology.orchestra.run.smithi112.stdout:4034 write 0xa82e3e1 thru 0xa832e5f (0x4a7f bytes) 2024-09-04T21:54:54.377 INFO:teuthology.orchestra.run.smithi112.stdout:4035 read 0xaa5792f thru 0xaa587c7 (0xe99 bytes) 2024-09-04T21:54:54.385 INFO:teuthology.orchestra.run.smithi112.stderr:set 2.7 pg_upmap_items mapping to [3->4,1->6] 2024-09-04T21:54:54.389 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:54.389+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:54.394 INFO:teuthology.orchestra.run.smithi112.stdout:4036 trunc from 0xc5c722c to 0x3a7c39d 2024-09-04T21:54:54.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:54:54.402 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:54.402 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:54.403 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:54.403 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:54.403 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:54.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:54.421 INFO:teuthology.orchestra.run.smithi112.stdout:4037 write 0x1c057cd thru 0x1c08ac5 (0x32f9 bytes) 2024-09-04T21:54:54.429 INFO:teuthology.orchestra.run.smithi112.stdout:4039 read 0x33e449a thru 0x33e86fe (0x4265 bytes) 2024-09-04T21:54:54.429 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.439 INFO:teuthology.orchestra.run.smithi112.stdout:4041 write 0x3cca132 thru 0x3cd74f2 (0xd3c1 bytes) 2024-09-04T21:54:54.448 INFO:teuthology.orchestra.run.smithi112.stdout:4042 writesame 0x3754992 thru 0x375a1f5 data_size 0x2(0x5864 bytes) 2024-09-04T21:54:54.473 INFO:teuthology.orchestra.run.smithi112.stdout:4043 trunc from 0x3cd74f3 to 0x9a941b7 2024-09-04T21:54:54.482 INFO:teuthology.orchestra.run.smithi112.stdout:4044 punch from 0x196613f to 0x196e3f3, (0x82b4 bytes) 2024-09-04T21:54:54.482 INFO:teuthology.orchestra.run.smithi112.stdout:4046 read 0xaabc79 thru 0xab104e (0x53d6 bytes) 2024-09-04T21:54:54.482 INFO:teuthology.orchestra.run.smithi112.stdout:4047 writesame 0x5992559 thru 0x5993898 data_size 0x4(0x1340 bytes) 2024-09-04T21:54:54.490 INFO:teuthology.orchestra.run.smithi112.stdout:4048 read 0x501b56e thru 0x501b9aa (0x43d bytes) 2024-09-04T21:54:54.490 INFO:teuthology.orchestra.run.smithi112.stdout:4049 punch from 0x2b3c265 to 0x2b47e66, (0xbc01 bytes) 2024-09-04T21:54:54.490 INFO:teuthology.orchestra.run.smithi112.stdout:4050 write 0x642f1d4 thru 0x643d64a (0xe477 bytes) 2024-09-04T21:54:54.490 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:54.485+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:54.497 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.497 INFO:teuthology.orchestra.run.smithi112.stdout:4052 read 0x4e17cbc thru 0x4e187fb (0xb40 bytes) 2024-09-04T21:54:54.510 INFO:teuthology.orchestra.run.smithi112.stdout:4053 write 0xc9c703a thru 0xc9c7678 (0x63f bytes) 2024-09-04T21:54:54.516 INFO:teuthology.orchestra.run.smithi112.stdout:4055 read 0xa6e1a38 thru 0xa6e8cfc (0x72c5 bytes) 2024-09-04T21:54:54.516 INFO:teuthology.orchestra.run.smithi112.stdout:4057 write 0x413dff2 thru 0x41436fd (0x570c bytes) 2024-09-04T21:54:54.525 INFO:teuthology.orchestra.run.smithi112.stdout:4058 write 0x856fafe thru 0x8573d53 (0x4256 bytes) 2024-09-04T21:54:54.534 INFO:teuthology.orchestra.run.smithi112.stdout:4059 read 0xa3d8fa7 thru 0xa3dbcf0 (0x2d4a bytes) 2024-09-04T21:54:54.534 INFO:teuthology.orchestra.run.smithi112.stdout:4061 read 0xbc3dcd4 thru 0xbc41f96 (0x42c3 bytes) 2024-09-04T21:54:54.534 INFO:teuthology.orchestra.run.smithi112.stdout:4062 write 0xa5fd56a thru 0xa608325 (0xadbc bytes) 2024-09-04T21:54:54.542 INFO:teuthology.orchestra.run.smithi112.stdout:4063 punch from 0x9548518 to 0x954c59a, (0x4082 bytes) 2024-09-04T21:54:54.542 INFO:teuthology.orchestra.run.smithi112.stdout:4065 punch from 0x57d3d59 to 0x57dda4b, (0x9cf2 bytes) 2024-09-04T21:54:54.551 INFO:teuthology.orchestra.run.smithi112.stdout:4066 writesame 0xcf43195 thru 0xcf464c5 data_size 0x1(0x3331 bytes) 2024-09-04T21:54:54.551 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.561 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.709047208999998, 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:54.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.582999946000001, 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.568 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:54.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:54.574 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.386446308000004, 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:54.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:54.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:54.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.359056809999998, 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:54.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:54.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.335792339000001, 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.586 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:54.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:54.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.308034548999998, 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.307918987000001, 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.600 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:54.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.007744201999998, 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:54.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:54.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:54.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.613 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.848611650000002, 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:54.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.619 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.848385592, 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:54.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:54.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.371172524000002, 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:54.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:54.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.141784413000003, 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.843593771999998, 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.843480917000001, 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:54.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:54.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.833819949000002, 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:54.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:54.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.813970224000002, 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:54.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.663741537999996, 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:54.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:54.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:54.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.997414632999998, 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:54.669 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:54.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.994275447, 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.675 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:54.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:54.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:54.680 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.975964457000003, 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:54.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:54.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:54.685 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:54.686 INFO:teuthology.orchestra.run.smithi112.stdout:4070 write 0xe452718 thru 0xe45f6f2 (0xcfdb bytes) 2024-09-04T21:54:54.686 INFO:teuthology.orchestra.run.smithi112.stdout:4072 read 0x2733d39 thru 0x273f0dd (0xb3a5 bytes) 2024-09-04T21:54:54.686 INFO:teuthology.orchestra.run.smithi112.stdout:4074 trunc from 0xe45f6f3 to 0x84b3adf 2024-09-04T21:54:54.687 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:54.689 INFO:teuthology.orchestra.run.smithi112.stdout:4075 read 0x36f6a18 thru 0x36f9e39 (0x3422 bytes) 2024-09-04T21:54:54.689 INFO:teuthology.orchestra.run.smithi112.stdout:4076 punch from 0x2aaa95d to 0x2ab5459, (0xaafc bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4077 punch from 0x6839a1f to 0x6843c2b, (0xa20c bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4078 read 0x6f80d16 thru 0x6f839f0 (0x2cdb bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4079 read 0x31925ef thru 0x3192b1f (0x531 bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4080 trunc from 0x84b3adf to 0x48a1a74 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4081 read 0x25f188 thru 0x26a046 (0xaebf bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4083 write 0x1ed80ec thru 0x1edbeda (0x3def bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4088 write 0x421ac63 thru 0x422398b (0x8d29 bytes) 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.690 INFO:teuthology.orchestra.run.smithi112.stdout:4091 trunc from 0x48a1a74 to 0xd0f95a9 2024-09-04T21:54:54.691 INFO:teuthology.orchestra.run.smithi112.stdout:4093 writesame 0xa082ace thru 0xa08d5fe data_size 0x5(0xab31 bytes) 2024-09-04T21:54:54.691 INFO:teuthology.orchestra.run.smithi112.stdout:4094 punch from 0xf8a447 to 0xf97644, (0xd1fd bytes) 2024-09-04T21:54:54.691 INFO:teuthology.orchestra.run.smithi112.stdout:4095 write 0x84c4843 thru 0x84cc4dc (0x7c9a bytes) 2024-09-04T21:54:54.691 INFO:teuthology.orchestra.run.smithi112.stdout:4096 write 0xbe22fa4 thru 0xbe28efc (0x5f59 bytes) 2024-09-04T21:54:54.691 INFO:teuthology.orchestra.run.smithi112.stdout:4097 trunc from 0xd0f95a9 to 0xd745591 2024-09-04T21:54:54.692 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:54.689+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:54.704 INFO:teuthology.orchestra.run.smithi112.stdout:4098 read 0x2c8f300 thru 0x2c9bd36 (0xca37 bytes) 2024-09-04T21:54:54.704 INFO:teuthology.orchestra.run.smithi112.stdout:4099 write 0x96a294a thru 0x96acba0 (0xa257 bytes) 2024-09-04T21:54:54.710 INFO:teuthology.orchestra.run.smithi112.stdout:4100 read 0x689318b thru 0x689d7ca (0xa640 bytes) 2024-09-04T21:54:54.722 INFO:teuthology.orchestra.run.smithi112.stdout:4101 write 0xdd2385b thru 0xdd258c6 (0x206c bytes) 2024-09-04T21:54:54.729 INFO:teuthology.orchestra.run.smithi112.stdout:4103 write 0xd49939f thru 0xd4a33e2 (0xa044 bytes) 2024-09-04T21:54:54.737 INFO:teuthology.orchestra.run.smithi112.stdout:4104 read 0xbe37f1c thru 0xbe3e02d (0x6112 bytes) 2024-09-04T21:54:54.738 INFO:teuthology.orchestra.run.smithi112.stdout:4106 write 0x8dc0a27 thru 0x8dc1337 (0x911 bytes) 2024-09-04T21:54:54.744 INFO:teuthology.orchestra.run.smithi112.stdout:4110 read 0x1295731 thru 0x1296852 (0x1122 bytes) 2024-09-04T21:54:54.744 INFO:teuthology.orchestra.run.smithi112.stdout:4111 write 0xd8916d7 thru 0xd898380 (0x6caa bytes) 2024-09-04T21:54:54.750 INFO:teuthology.orchestra.run.smithi112.stdout:4113 punch from 0x339b4ea to 0x339cd8b, (0x18a1 bytes) 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:4116 read 0x35d2b0a thru 0x35d5318 (0x280f bytes) 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:4118 punch from 0x61c0620 to 0x61cb4c4, (0xaea4 bytes) 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:4119 read 0xd450e8d thru 0xd45106b (0x1df bytes) 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:4120 read 0x65a76dd thru 0x65a9886 (0x21aa bytes) 2024-09-04T21:54:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:4121 write 0xa5eab82 thru 0xa5f93a2 (0xe821 bytes) 2024-09-04T21:54:54.759 INFO:teuthology.orchestra.run.smithi112.stdout:4122 write 0x137ee3a thru 0x13871ba (0x8381 bytes) 2024-09-04T21:54:54.769 INFO:teuthology.orchestra.run.smithi112.stdout:4123 write 0x7f574e7 thru 0x7f6583d (0xe357 bytes) 2024-09-04T21:54:54.779 INFO:teuthology.orchestra.run.smithi112.stdout:4124 write 0x8469c8e thru 0x847687c (0xcbef bytes) 2024-09-04T21:54:54.787 INFO:teuthology.orchestra.run.smithi112.stdout:4125 punch from 0x8cb0405 to 0x8cb6709, (0x6304 bytes) 2024-09-04T21:54:54.787 INFO:teuthology.orchestra.run.smithi112.stdout:4126 write 0xb52309 thru 0xb52494 (0x18c bytes) 2024-09-04T21:54:54.791 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:54.789+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:54.793 INFO:teuthology.orchestra.run.smithi112.stdout:4127 write 0x8ccbca5 thru 0x8cda244 (0xe5a0 bytes) 2024-09-04T21:54:54.802 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.802 INFO:teuthology.orchestra.run.smithi112.stdout:4130 write 0x3a487d2 thru 0x3a55663 (0xce92 bytes) 2024-09-04T21:54:54.822 INFO:teuthology.orchestra.run.smithi112.stdout:4131 trunc from 0xdd258c7 to 0x52eb58d 2024-09-04T21:54:54.828 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:54.828 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:54.828 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:54.829 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:54.844 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:54.880 INFO:teuthology.orchestra.run.smithi112.stdout:4132 write 0xdea0699 thru 0xdeab784 (0xb0ec bytes) 2024-09-04T21:54:54.887 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.892 INFO:teuthology.orchestra.run.smithi112.stdout:4134 trunc from 0xdeab785 to 0xb5852d3 2024-09-04T21:54:54.892 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:54.889+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:54.908 INFO:teuthology.orchestra.run.smithi112.stdout:4135 punch from 0x6420a04 to 0x642ec4f, (0xe24b bytes) 2024-09-04T21:54:54.908 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.908 INFO:teuthology.orchestra.run.smithi112.stdout:4140 punch from 0x8e30767 to 0x8e38cb6, (0x854f bytes) 2024-09-04T21:54:54.908 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:54.908 INFO:teuthology.orchestra.run.smithi112.stdout:4142 read 0xe5d4f4 thru 0xe6a0e3 (0xcbf0 bytes) 2024-09-04T21:54:54.908 INFO:teuthology.orchestra.run.smithi112.stdout:4143 read 0xb226dfe thru 0xb230d25 (0x9f28 bytes) 2024-09-04T21:54:54.923 INFO:teuthology.orchestra.run.smithi112.stdout:4144 writesame 0xe5c2051 thru 0xe5cbca5 data_size 0x1(0x9c55 bytes) 2024-09-04T21:54:54.923 INFO:teuthology.orchestra.run.smithi112.stdout:4145 punch from 0x843bb8b to 0x843ebf5, (0x306a bytes) 2024-09-04T21:54:54.923 INFO:teuthology.orchestra.run.smithi112.stdout:4147 read 0x68250ad thru 0x682b867 (0x67bb bytes) 2024-09-04T21:54:54.923 INFO:teuthology.orchestra.run.smithi112.stdout:4151 write 0x4bf13d7 thru 0x4bf7fbf (0x6be9 bytes) 2024-09-04T21:54:54.929 INFO:teuthology.orchestra.run.smithi112.stdout:4152 writesame 0x566eb54 thru 0x567632e data_size 0x1(0x77db bytes) 2024-09-04T21:54:54.949 INFO:teuthology.orchestra.run.smithi112.stdout:4153 write 0x2e9ef09 thru 0x2ea8efb (0x9ff3 bytes) 2024-09-04T21:54:54.958 INFO:teuthology.orchestra.run.smithi112.stdout:4155 write 0xc115232 thru 0xc115b2b (0x8fa bytes) 2024-09-04T21:54:54.963 INFO:teuthology.orchestra.run.smithi112.stdout:4157 write 0x6149dcb thru 0x61590fe (0xf334 bytes) 2024-09-04T21:54:54.972 INFO:teuthology.orchestra.run.smithi112.stdout:4160 read 0xb4bc8e0 thru 0xb4c7c7d (0xb39e bytes) 2024-09-04T21:54:54.984 INFO:teuthology.orchestra.run.smithi112.stdout:4161 trunc from 0xe5cbca6 to 0x61c05d8 2024-09-04T21:54:54.993 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:54.993+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:55.003 INFO:teuthology.orchestra.run.smithi112.stdout:4162 read 0x47890e8 thru 0x4793a10 (0xa929 bytes) 2024-09-04T21:54:55.004 INFO:teuthology.orchestra.run.smithi112.stdout:4163 read 0x3a095f3 thru 0x3a0b52a (0x1f38 bytes) 2024-09-04T21:54:55.004 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:55.004 INFO:teuthology.orchestra.run.smithi112.stdout:4165 write 0x1f055fc thru 0x1f066cc (0x10d1 bytes) 2024-09-04T21:54:55.011 INFO:teuthology.orchestra.run.smithi112.stdout:4166 read 0x18385d0 thru 0x1841aaa (0x94db bytes) 2024-09-04T21:54:55.012 INFO:teuthology.orchestra.run.smithi112.stdout:4167 read 0x46b6402 thru 0x46c3ce3 (0xd8e2 bytes) 2024-09-04T21:54:55.012 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:55.012 INFO:teuthology.orchestra.run.smithi112.stdout:4170 read 0x3c9b69c thru 0x3c9c48f (0xdf4 bytes) 2024-09-04T21:54:55.013 INFO:teuthology.orchestra.run.smithi112.stdout:4171 punch from 0x5cc313 to 0x5cd79d, (0x148a bytes) 2024-09-04T21:54:55.013 INFO:teuthology.orchestra.run.smithi112.stdout:4172 punch from 0x5c375ea to 0x5c40691, (0x90a7 bytes) 2024-09-04T21:54:55.018 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:55.018 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:55.019 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:55.019 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:55.019 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:55.037 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:55.038 INFO:teuthology.orchestra.run.smithi112.stdout:4173 writesame 0xbe8381b thru 0xbe85756 data_size 0x2(0x1f3c bytes) 2024-09-04T21:54:55.038 INFO:teuthology.orchestra.run.smithi112.stdout:4174 read 0xa3be1be thru 0xa3cd469 (0xf2ac bytes) 2024-09-04T21:54:55.038 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:55.044 INFO:teuthology.orchestra.run.smithi112.stdout:4177 trunc from 0xbe85757 to 0x75f8b6a 2024-09-04T21:54:55.054 INFO:teuthology.orchestra.run.smithi112.stdout:4179 read 0x5636336 thru 0x563f606 (0x92d1 bytes) 2024-09-04T21:54:55.055 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:55.055 INFO:teuthology.orchestra.run.smithi112.stdout:4182 punch from 0x6179823 to 0x6179e41, (0x61e bytes) 2024-09-04T21:54:55.055 INFO:teuthology.orchestra.run.smithi112.stdout:4184 read 0x29b7ba thru 0x2a0e20 (0x5667 bytes) 2024-09-04T21:54:55.062 INFO:teuthology.orchestra.run.smithi112.stdout:4185 trunc from 0x75f8b6a to 0x3253e95 2024-09-04T21:54:55.085 INFO:teuthology.orchestra.run.smithi112.stdout:4187 clone 21 order 22 su 65536 sc 10 2024-09-04T21:54:55.094 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:55.093+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:55.165 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:55.165 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:55.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:55.165 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32654 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 4800512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574878875, 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027786758000000002, 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.166 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32654 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588400+0000", 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3399999999999999e-07 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588403+0000", 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7249999999999999e-06 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588414+0000", 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0559e-05 2024-09-04T21:54:55.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588421+0000", 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5109999999999997e-06 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590927+0000", 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025057339999999999 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590933+0000", 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5820000000000001e-06 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610352+0000", 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.019419174000000001 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610442+0000", 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0125999999999997e-05 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.616186+0000", 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057437130000000001 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32670 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5062656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574265079, 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024152731, 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.171 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32670 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4199999999999995e-07 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589017+0000", 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.224e-06 2024-09-04T21:54:55.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589031+0000", 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4747e-05 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589039+0000", 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591456+0000", 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024168549999999999 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591462+0000", 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4979999999999997e-06 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613066+0000", 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0216037 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613120+0000", 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4209999999999998e-05 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613165+0000", 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5763999999999998e-05 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32671 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5079040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.176 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574225671000001, 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024282133000000001, 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32671 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589053+0000", 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589057+0000", 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589074+0000", 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7201999999999999e-05 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589083+0000", 2024-09-04T21:54:55.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5739999999999996e-06 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591487+0000", 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024049150000000001 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591493+0000", 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-06 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613189+0000", 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021695945000000001 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613260+0000", 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0905e-05 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613334+0000", 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4425999999999995e-05 2024-09-04T21:54:55.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32672 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5095424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574183068, 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024400369000000002, 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32672 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1299999999999999e-07 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589099+0000", 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.591e-06 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589115+0000", 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5591999999999999e-05 2024-09-04T21:54:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589128+0000", 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3210000000000001e-05 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591520+0000", 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023920529999999999 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591526+0000", 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613358+0000", 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021832949000000001 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613427+0000", 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8955999999999995e-05 2024-09-04T21:54:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613495+0000", 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7694999999999995e-05 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32673 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574138938000001, 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024527538000000002, 2024-09-04T21:54:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32673 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589140+0000", 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589143+0000", 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.331e-06 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589153+0000", 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7270000000000002e-06 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589161+0000", 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8149999999999995e-06 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591553+0000", 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023916469999999998 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591558+0000", 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1649999999999997e-06 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613520+0000", 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021961729999999999 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613591+0000", 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0735000000000006e-05 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613666+0000", 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5754999999999996e-05 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32674 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5128192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574105446000001, 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024653952999999999, 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.193 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32674 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589173+0000", 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000005e-07 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589186+0000", 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2972999999999999e-05 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589204+0000", 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7666999999999999e-05 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589212+0000", 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3999999999999992e-06 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591589+0000", 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023771370000000001 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591594+0000", 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3149999999999999e-06 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613690+0000", 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022095789000000001 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613759+0000", 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9142999999999999e-05 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613826+0000", 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6958999999999994e-05 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32675 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5144576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574053027000001, 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024775136, 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.198 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32675 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589226+0000", 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0500000000000003e-07 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589229+0000", 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999998e-06 2024-09-04T21:54:55.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589239+0000", 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589247+0000", 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0430000000000008e-06 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591622+0000", 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023745099999999998 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591627+0000", 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613851+0000", 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022223613 2024-09-04T21:54:55.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613923+0000", 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2203e-05 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614000+0000", 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32676 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5160960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.574019790000001, 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024914054000000001, 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32676 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589259+0000", 2024-09-04T21:54:55.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589276+0000", 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7686e-05 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589288+0000", 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1234e-05 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589296+0000", 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5769999999999994e-06 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591654+0000", 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023581769999999999 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591660+0000", 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3109999999999996e-06 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614025+0000", 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022365051 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614099+0000", 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3881000000000006e-05 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614172+0000", 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3498999999999995e-05 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.208 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32677 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5177344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.573961758999999, 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025050084, 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32677 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589317+0000", 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5300000000000003e-07 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589329+0000", 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1951e-05 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589346+0000", 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7108e-05 2024-09-04T21:54:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589354+0000", 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2630000000000003e-06 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591674+0000", 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023197980000000001 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591677+0000", 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5909999999999998e-06 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614205+0000", 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022528256999999999 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614287+0000", 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1886999999999997e-05 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614366+0000", 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9475999999999996e-05 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32678 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5193728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.214 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.573909567000001, 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025172158, 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32678 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589369+0000", 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8400000000000004e-07 2024-09-04T21:54:55.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589373+0000", 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589391+0000", 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8077e-05 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589400+0000", 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0790000000000007e-06 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591689+0000", 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002288815 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591692+0000", 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.666e-06 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614392+0000", 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022700287 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614462+0000", 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0468000000000004e-05 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614540+0000", 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8104999999999998e-05 2024-09-04T21:54:55.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32679 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5210112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.573865634000001, 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025302998, 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.220 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32679 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589413+0000", 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0100000000000004e-07 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589417+0000", 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7689999999999998e-06 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589435+0000", 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8185e-05 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589444+0000", 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4029999999999996e-06 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591704+0000", 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022601729999999999 2024-09-04T21:54:55.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591708+0000", 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.067e-06 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614571+0000", 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022863270000000002 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614640+0000", 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909e-05 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614715+0000", 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5520999999999997e-05 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32680 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5226496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.225 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.573820024, 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025450210000000001, 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32680 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589459+0000", 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-04T21:54:55.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589462+0000", 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7780000000000001e-06 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589479+0000", 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6651e-05 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589488+0000", 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6309999999999992e-06 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591737+0000", 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002249301 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591742+0000", 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4790000000000004e-06 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614747+0000", 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.023004539000000001 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614825+0000", 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8566000000000006e-05 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614908+0000", 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2558999999999993e-05 2024-09-04T21:54:55.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32929 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5120000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564204159999999, 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024146612000000001, 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32929 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:55.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599077+0000", 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.373e-06 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599083+0000", 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5060000000000001e-06 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599088+0000", 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0690000000000001e-06 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601461+0000", 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023728299999999998 2024-09-04T21:54:55.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601465+0000", 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5129999999999998e-06 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623133+0000", 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021667782 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623178+0000", 2024-09-04T21:54:55.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4950000000000002e-05 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623220+0000", 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2076999999999997e-05 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32930 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5136384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:55.236 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564182435999999, 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024238576000000001, 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32930 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:55.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599096+0000", 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599098+0000", 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2979999999999999e-06 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599105+0000", 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5479999999999998e-06 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599110+0000", 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1109999999999999e-06 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601491+0000", 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023813800000000002 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601496+0000", 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623247+0000", 2024-09-04T21:54:55.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021751177 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623291+0000", 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4184999999999998e-05 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623334+0000", 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2970999999999999e-05 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32931 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5152768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564160170000001, 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024330243000000001, 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32931 2024-09-04T21:54:55.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.61e-07 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599121+0000", 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.683e-06 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599129+0000", 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6359999999999994e-06 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599134+0000", 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5049999999999998e-06 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601519+0000", 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002384546 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601521+0000", 2024-09-04T21:54:55.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2919999999999998e-06 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623360+0000", 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021839164000000001 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623405+0000", 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4243999999999999e-05 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623448+0000", 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3411999999999997e-05 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32932 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5169152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564135136000001, 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024418507999999998, 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.247 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32932 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599144+0000", 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4799999999999997e-07 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599153+0000", 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9930000000000001e-06 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599161+0000", 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0900000000000005e-06 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599167+0000", 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601543+0000", 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023758540000000002 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601547+0000", 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1019999999999999e-06 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623474+0000", 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021927782 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623519+0000", 2024-09-04T21:54:55.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4263e-05 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623561+0000", 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2483e-05 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32933 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5185536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564101041000001, 2024-09-04T21:54:55.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024496766999999999, 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32933 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-07 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599180+0000", 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599188+0000", 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599203+0000", 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5099999999999999e-05 2024-09-04T21:54:55.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601573+0000", 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002369566 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601577+0000", 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5090000000000004e-06 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623587+0000", 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022009635999999999 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623631+0000", 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4605999999999999e-05 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623674+0000", 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2274000000000002e-05 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32934 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5201920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564064868999999, 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024572765999999999, 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32934 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599214+0000", 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599217+0000", 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.095e-06 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599224+0000", 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6760000000000007e-06 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599230+0000", 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8039999999999998e-06 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601603+0000", 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023727090000000002 2024-09-04T21:54:55.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601608+0000", 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-06 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623700+0000", 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022092219999999999 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623744+0000", 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4013e-05 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623786+0000", 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1749000000000003e-05 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32935 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5218304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:55.263 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564038584999999, 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024658065999999999, 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32935 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:55.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599240+0000", 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3500000000000002e-07 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599249+0000", 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599258+0000", 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7069999999999998e-06 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599264+0000", 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4790000000000001e-06 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601634+0000", 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023694020000000001 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601638+0000", 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623812+0000", 2024-09-04T21:54:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022173986 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623856+0000", 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3696000000000003e-05 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623897+0000", 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1530999999999999e-05 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32936 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5234688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.564003234000001, 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024740649, 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32936 2024-09-04T21:54:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2099999999999997e-07 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599278+0000", 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8870000000000001e-06 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599285+0000", 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4449999999999999e-06 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599291+0000", 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601668+0000", 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023769379999999999 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601673+0000", 2024-09-04T21:54:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8210000000000001e-06 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623927+0000", 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022253536000000001 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623972+0000", 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5216999999999998e-05 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.624015+0000", 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3519000000000001e-05 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.274 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:55.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:55.276 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:55.273+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:55.377 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:55.373+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:55.380 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:54:55.450 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:55.450 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:55.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:55.477 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:55.473+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:55.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:54:55.578 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:55.573+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:55.580 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-09-04T21:54:55.588 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:55.588 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:55.588 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:55.588 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:55.588 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:55.606 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:54:55.679 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:55.677+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:55.699 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:55.699 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:55.718 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:55.718 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:55.736 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:55.736 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:55.737 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 25.233887275000001, 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:54:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:54:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:54:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8891108889999995, 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:54:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:54:55.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8890822840000006, 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:54:55.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:54:55.751 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.889062247, 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:54:55.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8890430249999994, 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:54:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.760 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8890242340000007, 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:54:55.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8889922919999993, 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.765 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:54:55.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:54:55.769 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8889637629999996, 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:54:55.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8889441520000005, 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.774 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:54:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8889258590000004, 2024-09-04T21:54:55.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:54:55.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8889030449999993, 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:54:55.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:54:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8888839080000004, 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:54:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:54:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8888632419999993, 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:54:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:54:55.796 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8888324650000001, 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:54:55.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:54:55.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8888130969999999, 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:54:55.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:54:55.805 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8887891379999999, 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8887418890000003, 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.810 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8887162110000002, 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.814 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.818 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8886821850000004, 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:54:55.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8886577839999994, 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:55.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:54:55.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:54:55.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:55.827 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:55.829 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:54:55.829 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:55.829 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:55.830 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:55.830 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:55.830 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:55.831 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:55.831 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:55.831 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:55.825+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:55.841 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:55.841 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:55.930 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:55.925+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:55.961 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:55.961 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:55.975 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:54:55.987 INFO:teuthology.orchestra.run.smithi112.stdout:4188 write 0x41c41e8 thru 0x41d0c51 (0xca6a bytes) 2024-09-04T21:54:55.995 INFO:teuthology.orchestra.run.smithi112.stdout:4189 read 0x1aa329b thru 0x1aab152 (0x7eb8 bytes) 2024-09-04T21:54:55.995 INFO:teuthology.orchestra.run.smithi112.stdout:4190 write 0xcecd86 thru 0xcfa974 (0xdbef bytes) 2024-09-04T21:54:56.006 INFO:teuthology.orchestra.run.smithi112.stdout:4192 read 0x80ee53 thru 0x81d97f (0xeb2d bytes) 2024-09-04T21:54:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:4193 trunc from 0x41d0c52 to 0x72a1cb0 2024-09-04T21:54:56.025 INFO:teuthology.orchestra.run.smithi112.stdout:4194 write 0x78bc956 thru 0x78c577f (0x8e2a bytes) 2024-09-04T21:54:56.030 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:56.029+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:56.031 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.031 INFO:teuthology.orchestra.run.smithi112.stdout:4196 read 0x15c82f3 thru 0x15d0732 (0x8440 bytes) 2024-09-04T21:54:56.046 INFO:teuthology.orchestra.run.smithi112.stdout:4198 write 0xdb0fbce thru 0xdb1f410 (0xf843 bytes) 2024-09-04T21:54:56.051 INFO:teuthology.orchestra.run.smithi112.stdout:4199 write 0xa0557a5 thru 0xa060ff1 (0xb84d bytes) 2024-09-04T21:54:56.056 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.057 INFO:teuthology.orchestra.run.smithi112.stdout:4202 trunc from 0xdb1f411 to 0xd4f12f6 2024-09-04T21:54:56.068 INFO:teuthology.orchestra.run.smithi112.stdout:4203 read 0x1cc25eb thru 0x1cc3379 (0xd8f bytes) 2024-09-04T21:54:56.070 INFO:teuthology.orchestra.run.smithi112.stdout:4204 punch from 0x2ddfde9 to 0x2de20b5, (0x22cc bytes) 2024-09-04T21:54:56.070 INFO:teuthology.orchestra.run.smithi112.stdout:4206 read 0xb07dee3 thru 0xb07ecc8 (0xde6 bytes) 2024-09-04T21:54:56.070 INFO:teuthology.orchestra.run.smithi112.stdout:4207 write 0xa0d7655 thru 0xa0e4ae2 (0xd48e bytes) 2024-09-04T21:54:56.076 INFO:teuthology.orchestra.run.smithi112.stdout:4209 read 0x5e80267 thru 0x5e851f9 (0x4f93 bytes) 2024-09-04T21:54:56.089 INFO:teuthology.orchestra.run.smithi112.stdout:4210 write 0xedd5bac thru 0xeddf09d (0x94f2 bytes) 2024-09-04T21:54:56.091 INFO:teuthology.orchestra.run.smithi112.stdout:4211 write 0x95fb18e thru 0x96087cc (0xd63f bytes) 2024-09-04T21:54:56.097 INFO:teuthology.orchestra.run.smithi112.stdout:4213 writesame 0x9f223ac thru 0x9f277aa data_size 0x1(0x53ff bytes) 2024-09-04T21:54:56.113 INFO:teuthology.orchestra.run.smithi112.stdout:4215 read 0x769fbf1 thru 0x76a8baa (0x8fba bytes) 2024-09-04T21:54:56.113 INFO:teuthology.orchestra.run.smithi112.stdout:4216 write 0x8b8d1ed thru 0x8b96209 (0x901d bytes) 2024-09-04T21:54:56.118 INFO:teuthology.orchestra.run.smithi112.stdout:4218 read 0x65e316e thru 0x65ef2db (0xc16e bytes) 2024-09-04T21:54:56.118 INFO:teuthology.orchestra.run.smithi112.stdout:4220 write 0x31c208f thru 0x31d1e71 (0xfde3 bytes) 2024-09-04T21:54:56.127 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.128 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.128 INFO:teuthology.orchestra.run.smithi112.stdout:4227 read 0x553f4eb thru 0x554cea4 (0xd9ba bytes) 2024-09-04T21:54:56.128 INFO:teuthology.orchestra.run.smithi112.stdout:4228 punch from 0x68b7bbb to 0x68c57d5, (0xdc1a bytes) 2024-09-04T21:54:56.128 INFO:teuthology.orchestra.run.smithi112.stdout:4231 read 0x8ed4822 thru 0x8edafa1 (0x6780 bytes) 2024-09-04T21:54:56.128 INFO:teuthology.orchestra.run.smithi112.stdout:4232 write 0xec9d0de thru 0xeca49a8 (0x78cb bytes) 2024-09-04T21:54:56.130 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:56.125+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:54:56.134 INFO:teuthology.orchestra.run.smithi112.stdout:4233 read 0x6ed4e98 thru 0x6ee150e (0xc677 bytes) 2024-09-04T21:54:56.135 INFO:teuthology.orchestra.run.smithi112.stdout:4234 writesame 0x5124420 thru 0x512e906 data_size 0x1(0xa4e7 bytes) 2024-09-04T21:54:56.153 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:56.153 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:56.169 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:54:56.188 INFO:teuthology.orchestra.run.smithi112.stdout:4235 write 0xae66727 thru 0xae6796f (0x1249 bytes) 2024-09-04T21:54:56.194 INFO:teuthology.orchestra.run.smithi112.stdout:4236 read 0x9ccbcc5 thru 0x9ccfe33 (0x416f bytes) 2024-09-04T21:54:56.194 INFO:teuthology.orchestra.run.smithi112.stdout:4238 punch from 0xc17914e to 0xc17d6d3, (0x4585 bytes) 2024-09-04T21:54:56.204 INFO:teuthology.orchestra.run.smithi112.stdout:4239 trunc from 0xeddf09e to 0xae4f784 2024-09-04T21:54:56.227 INFO:teuthology.orchestra.run.smithi112.stdout:4240 read 0x11eaa72 thru 0x11f6d7b (0xc30a bytes) 2024-09-04T21:54:56.227 INFO:teuthology.orchestra.run.smithi112.stdout:4241 read 0xa46b0f3 thru 0xa475be3 (0xaaf1 bytes) 2024-09-04T21:54:56.228 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.231 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:56.229+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:56.256 INFO:teuthology.orchestra.run.smithi112.stdout:4244 trunc from 0xae4f784 to 0x14561b 2024-09-04T21:54:56.331 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:56.329+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:56.336 INFO:teuthology.orchestra.run.smithi112.stdout:4245 write 0x9ac8ea7 thru 0x9ace668 (0x57c2 bytes) 2024-09-04T21:54:56.341 INFO:teuthology.orchestra.run.smithi112.stdout:4246 read 0x23c2b23 thru 0x23cca6a (0x9f48 bytes) 2024-09-04T21:54:56.342 INFO:teuthology.orchestra.run.smithi112.stdout:4247 read 0x21e0011 thru 0x21eef8b (0xef7b bytes) 2024-09-04T21:54:56.344 INFO:teuthology.orchestra.run.smithi112.stdout:4248 trunc from 0x9ace669 to 0xa0b2ff6 2024-09-04T21:54:56.345 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:56.345 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:56.355 INFO:teuthology.orchestra.run.smithi112.stdout:4249 read 0xbbd1b thru 0xbff4e (0x4234 bytes) 2024-09-04T21:54:56.360 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:54:56.373 INFO:teuthology.orchestra.run.smithi112.stdout:4250 write 0xc3c10c3 thru 0xc3ca992 (0x98d0 bytes) 2024-09-04T21:54:56.379 INFO:teuthology.orchestra.run.smithi112.stdout:4251 write 0xa948d05 thru 0xa94d384 (0x4680 bytes) 2024-09-04T21:54:56.385 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.385 INFO:teuthology.orchestra.run.smithi112.stdout:4255 writesame 0x1699807 thru 0x169d909 data_size 0x11(0x4103 bytes) 2024-09-04T21:54:56.388 INFO:teuthology.orchestra.run.smithi112.stdout:4256 writesame 0x3e430e thru 0x3ec6e4 data_size 0x1(0x83d7 bytes) 2024-09-04T21:54:56.410 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.410 INFO:teuthology.orchestra.run.smithi112.stdout:4259 read 0x884706a thru 0x884e4cf (0x7466 bytes) 2024-09-04T21:54:56.429 INFO:teuthology.orchestra.run.smithi112.stdout:4260 write 0xe943df7 thru 0xe94f01b (0xb225 bytes) 2024-09-04T21:54:56.434 INFO:teuthology.orchestra.run.smithi112.stdout:4261 read 0x9051e00 thru 0x905b0e1 (0x92e2 bytes) 2024-09-04T21:54:56.434 INFO:teuthology.orchestra.run.smithi112.stdout:4263 writesame 0x1969017 thru 0x197159c data_size 0x51(0x8586 bytes) 2024-09-04T21:54:56.437 INFO:teuthology.orchestra.run.smithi112.stdout:4264 punch from 0x582f1ef to 0x583d28d, (0xe09e bytes) 2024-09-04T21:54:56.437 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:56.437 INFO:teuthology.orchestra.run.smithi112.stdout:4266 trunc from 0xe94f01c to 0xe897af3 2024-09-04T21:54:56.450 INFO:teuthology.orchestra.run.smithi112.stdout:4269 read 0xbdf54f9 thru 0xbe03305 (0xde0d bytes) 2024-09-04T21:54:56.450 INFO:teuthology.orchestra.run.smithi112.stdout:4270 read 0x19d4627 thru 0x19df759 (0xb133 bytes) 2024-09-04T21:54:56.450 INFO:teuthology.orchestra.run.smithi112.stdout:4271 punch from 0xaf4a025 to 0xaf4e181, (0x415c bytes) 2024-09-04T21:54:56.450 INFO:teuthology.orchestra.run.smithi112.stdout:4272 read 0x3ae2e3f thru 0x3ae8f71 (0x6133 bytes) 2024-09-04T21:54:56.451 INFO:teuthology.orchestra.run.smithi112.stdout:4274 clone 22 order 18 su 16384 sc 6 2024-09-04T21:54:56.451 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:56.449+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:56.525 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:56.525 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:56.533 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:56.529+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:56.539 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:54:56.634 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:56.633+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:56.706 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:56.706 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:56.720 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:54:56.767 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:56.765+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:56.835 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:56.833+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:56.842 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:56.842 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:56.856 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:54:56.936 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:56.933+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:57.023 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:57.023 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:57.037 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:54:57.037 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:57.033+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:57.138 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:57.133+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:57.160 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:57.160 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:57.174 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:54:57.263 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:57.261+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:57.340 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:57.337+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:57.345 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:57.345 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:57.359 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:54:57.407 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:54:57.447 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:57.445+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:57.505 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone21 from 0xe897af3 (overlap 0x14561b) to 0x82271 2024-09-04T21:54:57.541 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:57.541+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:57.542 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:57.543 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:57.560 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:54:57.595 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-09-04T21:54:57.647 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:57.645+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:57.688 INFO:teuthology.orchestra.run.smithi112.stdout:4276 write 0xb625f5a thru 0xb628b44 (0x2beb bytes) 2024-09-04T21:54:57.705 INFO:teuthology.orchestra.run.smithi112.stdout:4277 write 0xcb84d25 thru 0xcb8ed22 (0x9ffe bytes) 2024-09-04T21:54:57.719 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:57.719 INFO:teuthology.orchestra.run.smithi112.stdout:4281 write 0x542176e thru 0x542ebff (0xd492 bytes) 2024-09-04T21:54:57.734 INFO:teuthology.orchestra.run.smithi112.stdout:4284 write 0x428e3d thru 0x4295d1 (0x795 bytes) 2024-09-04T21:54:57.736 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:57.737 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:57.743 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:57.741+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:57.743 INFO:teuthology.orchestra.run.smithi112.stdout:4286 read 0xbeca2fb thru 0xbecaac3 (0x7c9 bytes) 2024-09-04T21:54:57.744 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:57.744 INFO:teuthology.orchestra.run.smithi112.stdout:4291 read 0xe1d1976 thru 0xe1daa8f (0x911a bytes) 2024-09-04T21:54:57.744 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:57.744 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:57.745 INFO:teuthology.orchestra.run.smithi112.stdout:4297 write 0xaced231 thru 0xacfa1e3 (0xcfb3 bytes) 2024-09-04T21:54:57.754 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:54:57.771 INFO:teuthology.orchestra.run.smithi112.stdout:4298 trunc from 0xe897af3 to 0x7111c1f 2024-09-04T21:54:57.804 INFO:teuthology.orchestra.run.smithi112.stdout:4300 flatten 2024-09-04T21:54:57.844 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:57.841+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:57.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:54:57.944 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:57.941+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:57.982 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:54:57.982 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:54:57.998 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:54:58.007 INFO:teuthology.orchestra.run.smithi112.stdout:4301 writesame 0xdbdac29 thru 0xdbe26a0 data_size 0x4(0x7a78 bytes) 2024-09-04T21:54:58.007 INFO:teuthology.orchestra.run.smithi112.stdout:4303 clone 23 order 25 su 262144 sc 7 2024-09-04T21:54:58.014 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:54:58.014 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:54:58.031 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:54:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:54:58.046 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:58.041+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:54:58.048 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:54:58.066 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:54:58.066 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:54:58.084 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:54:58.084 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:54:58.105 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:54:58.105 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:54:58.115 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:58.115 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:58.115 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:58.115 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:58.126 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:54:58.126 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:54:58.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:54:58.147 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:54:58.147 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:54:58.148 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:58.145+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:58.248 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:58.245+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:58.273 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:58.273 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:58.273 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:58.273 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:58.274 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:58.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:54:58.349 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:58.345+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:58.407 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:54:58.449 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:58.445+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:58.459 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:58.459 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:58.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:58.459 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:58.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.459 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783175757999999, 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:54:58.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:54:58.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:58.464 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783148973000003, 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:54:58.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:54:58.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:58.469 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.78312004, 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:54:58.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783096393000001, 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:54:58.474 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:54:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783076612000002, 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:58.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783057217, 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.484 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:54:58.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783037944, 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.489 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:54:58.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.493 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.783018722000001, 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.494 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:58.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:54:58.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782999885000002, 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:54:58.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:54:58.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782980481999999, 2024-09-04T21:54:58.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:58.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:54:58.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:54:58.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.508 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782959425999998, 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:54:58.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.513 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782927248, 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.514 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:54:58.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782901027000001, 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:54:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782872818000001, 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:54:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:54:58.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782852777999999, 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:54:58.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782833396000001, 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:54:58.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:54:58.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:58.537 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782809808000003, 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:54:58.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782789491000003, 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.542 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782769938999998, 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.547 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:54:58.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.782750180000001, 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:54:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:58.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:58.556 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:58.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:54:58.558 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:58.557+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:58.659 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:58.657+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:58.731 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:58.731 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:58.731 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:58.731 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:58.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:54:58.760 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:58.757+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:58.792 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-09-04T21:54:58.798 INFO:teuthology.orchestra.run.smithi112.stdout:4304 punch from 0xb757065 to 0xb75a8a3, (0x383e bytes) 2024-09-04T21:54:58.805 INFO:teuthology.orchestra.run.smithi112.stdout:4305 punch from 0xa1e207 to 0xa1fef1, (0x1cea bytes) 2024-09-04T21:54:58.805 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:58.821 INFO:teuthology.orchestra.run.smithi112.stdout:4307 trunc from 0xdbe26a1 to 0x4e7691d 2024-09-04T21:54:58.860 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:58.857+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:58.870 INFO:teuthology.orchestra.run.smithi112.stdout:4308 write 0x249970e thru 0x24a77ca (0xe0bd bytes) 2024-09-04T21:54:58.874 INFO:teuthology.orchestra.run.smithi112.stdout:4309 write 0x346d59b thru 0x3471851 (0x42b7 bytes) 2024-09-04T21:54:58.897 INFO:teuthology.orchestra.run.smithi112.stdout:4311 write 0x964d4fb thru 0x964f2bf (0x1dc5 bytes) 2024-09-04T21:54:58.900 INFO:teuthology.orchestra.run.smithi112.stdout:4312 punch from 0x18c837c to 0x18c9ee5, (0x1b69 bytes) 2024-09-04T21:54:58.919 INFO:teuthology.orchestra.run.smithi112.stdout:4313 writesame 0xb9e9b0a thru 0xb9f32ad data_size 0x1e(0x97a4 bytes) 2024-09-04T21:54:58.921 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout:4315 punch from 0x2b4b232 to 0x2b4eca2, (0x3a70 bytes) 2024-09-04T21:54:58.922 INFO:teuthology.orchestra.run.smithi112.stdout:4316 read 0xb7f5cb8 thru 0xb7fbfda (0x6323 bytes) 2024-09-04T21:54:58.929 INFO:teuthology.orchestra.run.smithi112.stdout:4317 trunc from 0xb9f32ae to 0x8e43d8a 2024-09-04T21:54:58.940 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:54:58.947 INFO:teuthology.orchestra.run.smithi112.stdout:4318 writesame 0x88245b8 thru 0x8831ab8 data_size 0x1(0xd501 bytes) 2024-09-04T21:54:58.983 INFO:teuthology.orchestra.run.smithi112.stdout:4320 write 0x24ad8eb thru 0x24ba954 (0xd06a bytes) 2024-09-04T21:54:58.984 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:58.981+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:58.995 INFO:teuthology.orchestra.run.smithi112.stdout:4321 trunc from 0x8e43d8a to 0xe8c0969 2024-09-04T21:54:59.006 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.025 INFO:teuthology.orchestra.run.smithi112.stdout:4323 trunc from 0xe8c0969 to 0x19dc396 2024-09-04T21:54:59.062 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:54:59.061+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:54:59.065 INFO:teuthology.orchestra.run.smithi112.stdout:4324 writesame 0x732848b thru 0x732917b data_size 0x1(0xcf1 bytes) 2024-09-04T21:54:59.068 INFO:teuthology.orchestra.run.smithi112.stdout:4325 punch from 0x2de55ff to 0x2de9a55, (0x4456 bytes) 2024-09-04T21:54:59.076 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.222102989, 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:54:59.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:54:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.096055726000003, 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:54:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.899502087999998, 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.095 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.87211259, 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:54:59.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:54:59.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.848848119000003, 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:54:59.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.821090329, 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.107 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:54:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:54:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.820974767000003, 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:54:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.520799982, 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.120 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:54:59.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:54:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 52.361667429999997, 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:54:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:54:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 52.361441372000002, 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.132 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:54:59.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.884228303999997, 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:54:59.138 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:54:59.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:54:59.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:59.144 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.654840192999998, 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:54:59.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:54:59.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:59.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:54:59.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.356649552, 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.151 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:54:59.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:54:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.356536697000003, 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.157 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:54:59.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:54:59.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.346875728999997, 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:54:59.163 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:54:59.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:54:59.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.327026003999997, 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.168 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:54:59.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.176797317999998, 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:54:59.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:54:59.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:54:59.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:59.179 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.510470413, 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:54:59.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:54:59.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.507331227000002, 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:54:59.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.489020236999998, 2024-09-04T21:54:59.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:54:59.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:54:59.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.195 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:59.196 INFO:teuthology.orchestra.run.smithi112.stdout:4326 write 0x99c9bce thru 0x99cd082 (0x34b5 bytes) 2024-09-04T21:54:59.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:54:59.198 INFO:teuthology.orchestra.run.smithi112.stdout:4329 punch from 0x2ca1714 to 0x2cb02fe, (0xebea bytes) 2024-09-04T21:54:59.198 INFO:teuthology.orchestra.run.smithi112.stdout:4330 trunc from 0x99cd083 to 0x136a5c8 2024-09-04T21:54:59.198 INFO:teuthology.orchestra.run.smithi112.stdout:4331 writesame 0x728b494 thru 0x7297c47 data_size 0x2(0xc7b4 bytes) 2024-09-04T21:54:59.198 INFO:teuthology.orchestra.run.smithi112.stdout:4332 read 0x2c6f316 thru 0x2c73526 (0x4211 bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4335 write 0x6123592 thru 0x612ee61 (0xb8d0 bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4337 write 0x2f5bba0 thru 0x2f6848d (0xc8ee bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4339 punch from 0x1d35643 to 0x1d3ddd6, (0x8793 bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4341 punch from 0x39ca633 to 0x39cccb9, (0x2686 bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4342 read 0x6b21444 thru 0x6b2bb64 (0xa721 bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4343 write 0x65be096 thru 0x65cc9b7 (0xe922 bytes) 2024-09-04T21:54:59.199 INFO:teuthology.orchestra.run.smithi112.stdout:4345 trunc from 0x7297c48 to 0x505bf6e 2024-09-04T21:54:59.199 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:54:59.197+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:54:59.225 INFO:teuthology.orchestra.run.smithi112.stdout:4346 writesame 0x96aee48 thru 0x96b4177 data_size 0x2c(0x5330 bytes) 2024-09-04T21:54:59.228 INFO:teuthology.orchestra.run.smithi112.stdout:4347 read 0x3320fbe thru 0x332e32c (0xd36f bytes) 2024-09-04T21:54:59.229 INFO:teuthology.orchestra.run.smithi112.stdout:4349 write 0x5993a5b thru 0x599f1e9 (0xb78f bytes) 2024-09-04T21:54:59.234 INFO:teuthology.orchestra.run.smithi112.stdout:4350 writesame 0x49e1734 thru 0x49ed617 data_size 0x26(0xbee4 bytes) 2024-09-04T21:54:59.238 INFO:teuthology.orchestra.run.smithi112.stdout:4352 punch from 0x6974af3 to 0x697684a, (0x1d57 bytes) 2024-09-04T21:54:59.238 INFO:teuthology.orchestra.run.smithi112.stdout:4353 read 0x769a252 thru 0x76a23fd (0x81ac bytes) 2024-09-04T21:54:59.262 INFO:teuthology.orchestra.run.smithi112.stdout:4354 trunc from 0x96b4178 to 0x3a5d6c 2024-09-04T21:54:59.282 INFO:teuthology.orchestra.run.smithi112.stdout:4355 read 0x255b42 thru 0x25a6dc (0x4b9b bytes) 2024-09-04T21:54:59.294 INFO:teuthology.orchestra.run.smithi112.stdout:4356 trunc from 0x3a5d6c to 0x77f3bb0 2024-09-04T21:54:59.297 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:59.297+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:59.327 INFO:teuthology.orchestra.run.smithi112.stdout:4357 writesame 0xb72effb thru 0xb734eb8 data_size 0x43(0x5ebe bytes) 2024-09-04T21:54:59.330 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.331 INFO:teuthology.orchestra.run.smithi112.stdout:4359 write 0xb4030fa thru 0xb40b67c (0x8583 bytes) 2024-09-04T21:54:59.334 INFO:teuthology.orchestra.run.smithi112.stdout:4360 read 0x21b2b90 thru 0x21c2107 (0xf578 bytes) 2024-09-04T21:54:59.340 INFO:teuthology.orchestra.run.smithi112.stdout:4361 trunc from 0xb734eb9 to 0xce846aa 2024-09-04T21:54:59.352 INFO:teuthology.orchestra.run.smithi112.stdout:4362 trunc from 0xce846aa to 0xd0bdc6d 2024-09-04T21:54:59.361 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:59.361 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:59.365 INFO:teuthology.orchestra.run.smithi112.stdout:4363 write 0x7eb3482 thru 0x7eb8f00 (0x5a7f bytes) 2024-09-04T21:54:59.370 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.370 INFO:teuthology.orchestra.run.smithi112.stdout:4365 writesame 0x510f8f6 thru 0x511a60b data_size 0x69(0xad16 bytes) 2024-09-04T21:54:59.374 INFO:teuthology.orchestra.run.smithi112.stdout:4366 read 0xb60113f thru 0xb60791e (0x67e0 bytes) 2024-09-04T21:54:59.375 INFO:teuthology.orchestra.run.smithi112.stdout:4367 write 0x5ac622d thru 0x5acf790 (0x9564 bytes) 2024-09-04T21:54:59.376 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:54:59.378 INFO:teuthology.orchestra.run.smithi112.stdout:4368 punch from 0xc474c8c to 0xc47d859, (0x8bcd bytes) 2024-09-04T21:54:59.396 INFO:teuthology.orchestra.run.smithi112.stdout:4369 write 0xe8f8fcc thru 0xe902ad6 (0x9b0b bytes) 2024-09-04T21:54:59.397 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:54:59.397+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:54:59.405 INFO:teuthology.orchestra.run.smithi112.stdout:4370 read 0xaa0c2ec thru 0xaa0fb6b (0x3880 bytes) 2024-09-04T21:54:59.406 INFO:teuthology.orchestra.run.smithi112.stdout:4372 read 0xd568670 thru 0xd56b7de (0x316f bytes) 2024-09-04T21:54:59.407 INFO:teuthology.orchestra.run.smithi112.stdout:4375 punch from 0x3086db5 to 0x30955ca, (0xe815 bytes) 2024-09-04T21:54:59.407 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.407 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.407 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.407 INFO:teuthology.orchestra.run.smithi112.stdout:4380 punch from 0xe039969 to 0xe047381, (0xda18 bytes) 2024-09-04T21:54:59.407 INFO:teuthology.orchestra.run.smithi112.stdout:4381 writesame 0x31e5f02 thru 0x31e5ff7 data_size 0x3(0xf6 bytes) 2024-09-04T21:54:59.410 INFO:teuthology.orchestra.run.smithi112.stdout:4383 write 0x8ca5e73 thru 0x8cacfc2 (0x7150 bytes) 2024-09-04T21:54:59.414 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7] out_osds: [1, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:54:59.414 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:54:59.414 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-04T21:54:59.414 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-04T21:54:59.414 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-04T21:54:59.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-04T21:54:59.415 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.415 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.415 INFO:teuthology.orchestra.run.smithi112.stdout:4389 read 0x1849e39 thru 0x185743b (0xd603 bytes) 2024-09-04T21:54:59.416 INFO:teuthology.orchestra.run.smithi112.stdout:4390 punch from 0x23b90d2 to 0x23c0d5e, (0x7c8c bytes) 2024-09-04T21:54:59.434 INFO:teuthology.orchestra.run.smithi112.stdout:4393 trunc from 0xe902ad7 to 0x69c5baf 2024-09-04T21:54:59.463 INFO:teuthology.orchestra.run.smithi112.stdout:4394 read 0x29dc27a thru 0x29e93c9 (0xd150 bytes) 2024-09-04T21:54:59.464 INFO:teuthology.orchestra.run.smithi112.stdout:4395 write 0x136ad40 thru 0x13736fa (0x89bb bytes) 2024-09-04T21:54:59.467 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.480 INFO:teuthology.orchestra.run.smithi112.stdout:4397 writesame 0x6c6d6cf thru 0x6c7877a data_size 0x6(0xb0ac bytes) 2024-09-04T21:54:59.483 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.484 INFO:teuthology.orchestra.run.smithi112.stdout:4399 read 0x147af32 thru 0x148381d (0x88ec bytes) 2024-09-04T21:54:59.485 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.499 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:54:59.497+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:54:59.508 INFO:teuthology.orchestra.run.smithi112.stdout:4403 write 0xba008a9 thru 0xba02626 (0x1d7e bytes) 2024-09-04T21:54:59.509 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:54:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:54:59.510 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:59.511 INFO:teuthology.orchestra.run.smithi112.stdout:4404 writesame 0x7bf0281 thru 0x7bfa8f1 data_size 0x3(0xa671 bytes) 2024-09-04T21:54:59.520 INFO:teuthology.orchestra.run.smithi112.stdout:4405 writesame 0x7e7277f thru 0x7e7d368 data_size 0x36(0xabea bytes) 2024-09-04T21:54:59.525 INFO:teuthology.orchestra.run.smithi112.stdout:4406 write 0xabb09a2 thru 0xabbc229 (0xb888 bytes) 2024-09-04T21:54:59.526 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:54:59.529 INFO:teuthology.orchestra.run.smithi112.stdout:4408 read 0x466cb26 thru 0x466cb94 (0x6f bytes) 2024-09-04T21:54:59.531 INFO:teuthology.orchestra.run.smithi112.stdout:4409 punch from 0x2c26c3a to 0x2c28334, (0x16fa bytes) 2024-09-04T21:54:59.532 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.532 INFO:teuthology.orchestra.run.smithi112.stdout:4412 writesame 0x1656244 thru 0x16639f0 data_size 0x1(0xd7ad bytes) 2024-09-04T21:54:59.546 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-04T21:54:59.547 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:59.567 INFO:teuthology.orchestra.run.smithi112.stdout:4413 read 0x71fe619 thru 0x720177e (0x3166 bytes) 2024-09-04T21:54:59.569 INFO:teuthology.orchestra.run.smithi112.stdout:4414 read 0x4d2c1be thru 0x4d3312d (0x6f70 bytes) 2024-09-04T21:54:59.569 INFO:teuthology.orchestra.run.smithi112.stdout:4415 read 0x403792 thru 0x40af87 (0x77f6 bytes) 2024-09-04T21:54:59.570 INFO:teuthology.orchestra.run.smithi112.stdout:4416 punch from 0x54ac9f to 0x550eb5, (0x6216 bytes) 2024-09-04T21:54:59.571 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.571 INFO:teuthology.orchestra.run.smithi112.stdout:4418 read 0x9c5cd7c thru 0x9c68bf2 (0xbe77 bytes) 2024-09-04T21:54:59.571 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.571 INFO:teuthology.orchestra.run.smithi112.stdout:4422 punch from 0xb567839 to 0xb57352b, (0xbcf2 bytes) 2024-09-04T21:54:59.574 INFO:teuthology.orchestra.run.smithi112.stdout:4423 trunc from 0xba02627 to 0x9cedff3 2024-09-04T21:54:59.590 INFO:teuthology.orchestra.run.smithi112.stdout:4425 write 0x85d022 thru 0x867dc6 (0xada5 bytes) 2024-09-04T21:54:59.594 INFO:teuthology.orchestra.run.smithi112.stdout:4426 punch from 0x8745abc to 0x875201a, (0xc55e bytes) 2024-09-04T21:54:59.594 INFO:teuthology.orchestra.run.smithi112.stdout:4427 writesame 0x8a89767 thru 0x8a94b92 data_size 0x1a(0xb42c bytes) 2024-09-04T21:54:59.599 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:54:59.597+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:54:59.613 INFO:teuthology.orchestra.run.smithi112.stdout:4429 write 0xc893bea thru 0xc89c45e (0x8875 bytes) 2024-09-04T21:54:59.617 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.617 INFO:teuthology.orchestra.run.smithi112.stdout:4432 write 0x683f875 thru 0x6842e54 (0x35e0 bytes) 2024-09-04T21:54:59.621 INFO:teuthology.orchestra.run.smithi112.stdout:4433 read 0xa105ce2 thru 0xa114523 (0xe842 bytes) 2024-09-04T21:54:59.621 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.622 INFO:teuthology.orchestra.run.smithi112.stdout:4435 writesame 0x9008fde thru 0x90128cf data_size 0x1(0x98f2 bytes) 2024-09-04T21:54:59.653 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32654 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 4800512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.063300794, 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027786758000000002, 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.654 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32654 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588400+0000", 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3399999999999999e-07 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588403+0000", 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7249999999999999e-06 2024-09-04T21:54:59.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588414+0000", 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0559e-05 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588421+0000", 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5109999999999997e-06 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590927+0000", 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025057339999999999 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590933+0000", 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5820000000000001e-06 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610352+0000", 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.019419174000000001 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610442+0000", 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0125999999999997e-05 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.616186+0000", 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057437130000000001 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.659 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32670 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5062656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062686998, 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024152731, 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32670 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:54:59.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4199999999999995e-07 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589017+0000", 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.224e-06 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589031+0000", 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4747e-05 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589039+0000", 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591456+0000", 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024168549999999999 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591462+0000", 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4979999999999997e-06 2024-09-04T21:54:59.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613066+0000", 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0216037 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613120+0000", 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4209999999999998e-05 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613165+0000", 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5763999999999998e-05 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32671 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5079040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062647590000001, 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024282133000000001, 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.665 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32671 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589053+0000", 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589057+0000", 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589074+0000", 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7201999999999999e-05 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589083+0000", 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5739999999999996e-06 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591487+0000", 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024049150000000001 2024-09-04T21:54:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591493+0000", 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-06 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613189+0000", 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021695945000000001 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613260+0000", 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0905e-05 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613334+0000", 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4425999999999995e-05 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.670 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32672 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5095424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062604987, 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024400369000000002, 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32672 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1299999999999999e-07 2024-09-04T21:54:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589099+0000", 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.591e-06 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589115+0000", 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5591999999999999e-05 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589128+0000", 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3210000000000001e-05 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591520+0000", 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023920529999999999 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591526+0000", 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613358+0000", 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021832949000000001 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613427+0000", 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8955999999999995e-05 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613495+0000", 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7694999999999995e-05 2024-09-04T21:54:59.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32673 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062560857000001, 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024527538000000002, 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.676 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32673 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:54:59.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589140+0000", 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589143+0000", 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.331e-06 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589153+0000", 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7270000000000002e-06 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589161+0000", 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8149999999999995e-06 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591553+0000", 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023916469999999998 2024-09-04T21:54:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591558+0000", 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1649999999999997e-06 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613520+0000", 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021961729999999999 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613591+0000", 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0735000000000006e-05 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613666+0000", 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5754999999999996e-05 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32674 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5128192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:59.681 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062527365000001, 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024653952999999999, 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32674 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:59.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589173+0000", 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000005e-07 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589186+0000", 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2972999999999999e-05 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589204+0000", 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7666999999999999e-05 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589212+0000", 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3999999999999992e-06 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591589+0000", 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023771370000000001 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591594+0000", 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3149999999999999e-06 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613690+0000", 2024-09-04T21:54:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022095789000000001 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613759+0000", 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9142999999999999e-05 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613826+0000", 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6958999999999994e-05 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32675 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5144576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062474945999998, 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024775136, 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32675 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:54:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589226+0000", 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0500000000000003e-07 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589229+0000", 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999998e-06 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589239+0000", 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589247+0000", 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0430000000000008e-06 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591622+0000", 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023745099999999998 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591627+0000", 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613851+0000", 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022223613 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613923+0000", 2024-09-04T21:54:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2203e-05 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614000+0000", 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32676 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5160960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062441709000002, 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024914054000000001, 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32676 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589259+0000", 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589276+0000", 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7686e-05 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589288+0000", 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1234e-05 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589296+0000", 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5769999999999994e-06 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591654+0000", 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023581769999999999 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591660+0000", 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3109999999999996e-06 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614025+0000", 2024-09-04T21:54:59.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022365051 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614099+0000", 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3881000000000006e-05 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614172+0000", 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3498999999999995e-05 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32677 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5177344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062383678, 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025050084, 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.698 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32677 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589317+0000", 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5300000000000003e-07 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589329+0000", 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1951e-05 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589346+0000", 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7108e-05 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589354+0000", 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2630000000000003e-06 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591674+0000", 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023197980000000001 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591677+0000", 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5909999999999998e-06 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614205+0000", 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022528256999999999 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614287+0000", 2024-09-04T21:54:59.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1886999999999997e-05 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614366+0000", 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9475999999999996e-05 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32678 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5193728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:59.703 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062331486000001, 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025172158, 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32678 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589369+0000", 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8400000000000004e-07 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589373+0000", 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589391+0000", 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8077e-05 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589400+0000", 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0790000000000007e-06 2024-09-04T21:54:59.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591689+0000", 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002288815 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591692+0000", 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.666e-06 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614392+0000", 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022700287 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614462+0000", 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0468000000000004e-05 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614540+0000", 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8104999999999998e-05 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.708 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32679 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5210112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062287553000001, 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025302998, 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.709 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32679 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589413+0000", 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0100000000000004e-07 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589417+0000", 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7689999999999998e-06 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589435+0000", 2024-09-04T21:54:59.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8185e-05 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589444+0000", 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4029999999999996e-06 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591704+0000", 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022601729999999999 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591708+0000", 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.067e-06 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614571+0000", 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022863270000000002 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614640+0000", 2024-09-04T21:54:59.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909e-05 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614715+0000", 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5520999999999997e-05 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32680 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5226496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.062241943, 2024-09-04T21:54:59.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025450210000000001, 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32680 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589459+0000", 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589462+0000", 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7780000000000001e-06 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589479+0000", 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6651e-05 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589488+0000", 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6309999999999992e-06 2024-09-04T21:54:59.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591737+0000", 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002249301 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591742+0000", 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4790000000000004e-06 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614747+0000", 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.023004539000000001 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614825+0000", 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8566000000000006e-05 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614908+0000", 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2558999999999993e-05 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.719 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32929 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5120000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052626078999999, 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024146612000000001, 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.720 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32929 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599077+0000", 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.373e-06 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599083+0000", 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5060000000000001e-06 2024-09-04T21:54:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599088+0000", 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0690000000000001e-06 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601461+0000", 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023728299999999998 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601465+0000", 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5129999999999998e-06 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623133+0000", 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021667782 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623178+0000", 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4950000000000002e-05 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623220+0000", 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2076999999999997e-05 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32930 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5136384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052604355, 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024238576000000001, 2024-09-04T21:54:59.725 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32930 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599096+0000", 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599098+0000", 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2979999999999999e-06 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599105+0000", 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5479999999999998e-06 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599110+0000", 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1109999999999999e-06 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601491+0000", 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023813800000000002 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601496+0000", 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623247+0000", 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021751177 2024-09-04T21:54:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623291+0000", 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4184999999999998e-05 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623334+0000", 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2970999999999999e-05 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32931 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5152768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052582089000001, 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024330243000000001, 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32931 2024-09-04T21:54:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.61e-07 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599121+0000", 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.683e-06 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599129+0000", 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6359999999999994e-06 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599134+0000", 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5049999999999998e-06 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601519+0000", 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002384546 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601521+0000", 2024-09-04T21:54:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2919999999999998e-06 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623360+0000", 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021839164000000001 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623405+0000", 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4243999999999999e-05 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623448+0000", 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3411999999999997e-05 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32932 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5169152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052557055000001, 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024418507999999998, 2024-09-04T21:54:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32932 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599144+0000", 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4799999999999997e-07 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599153+0000", 2024-09-04T21:54:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9930000000000001e-06 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599161+0000", 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0900000000000005e-06 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599167+0000", 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601543+0000", 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023758540000000002 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601547+0000", 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1019999999999999e-06 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623474+0000", 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021927782 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623519+0000", 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4263e-05 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623561+0000", 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2483e-05 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32933 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5185536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052522960000001, 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024496766999999999, 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32933 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:54:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-07 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599180+0000", 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599188+0000", 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599203+0000", 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5099999999999999e-05 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601573+0000", 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002369566 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601577+0000", 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5090000000000004e-06 2024-09-04T21:54:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623587+0000", 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022009635999999999 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623631+0000", 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4605999999999999e-05 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623674+0000", 2024-09-04T21:54:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2274000000000002e-05 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32934 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5201920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052486788, 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024572765999999999, 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32934 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599214+0000", 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599217+0000", 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.095e-06 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599224+0000", 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6760000000000007e-06 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599230+0000", 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8039999999999998e-06 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601603+0000", 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023727090000000002 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601608+0000", 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-06 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623700+0000", 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022092219999999999 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623744+0000", 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4013e-05 2024-09-04T21:54:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623786+0000", 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1749000000000003e-05 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32935 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5218304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052460503999999, 2024-09-04T21:54:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024658065999999999, 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32935 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599240+0000", 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3500000000000002e-07 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599249+0000", 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599258+0000", 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7069999999999998e-06 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599264+0000", 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4790000000000001e-06 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601634+0000", 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023694020000000001 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601638+0000", 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623812+0000", 2024-09-04T21:54:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022173986 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623856+0000", 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3696000000000003e-05 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623897+0000", 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1530999999999999e-05 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32936 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5234688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.052425153000002, 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024740649, 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32936 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:54:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2099999999999997e-07 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599278+0000", 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8870000000000001e-06 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599285+0000", 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4449999999999999e-06 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599291+0000", 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601668+0000", 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023769379999999999 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601673+0000", 2024-09-04T21:54:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8210000000000001e-06 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623927+0000", 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022253536000000001 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623972+0000", 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5216999999999998e-05 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:54:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.624015+0000", 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3519000000000001e-05 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:54:59.763 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:59.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:54:59.765 INFO:teuthology.orchestra.run.smithi112.stdout:4436 write 0xdfdc211 thru 0xdfe3b04 (0x78f4 bytes) 2024-09-04T21:54:59.765 INFO:teuthology.orchestra.run.smithi112.stdout:4438 read 0xd6c7456 thru 0xd6d11a7 (0x9d52 bytes) 2024-09-04T21:54:59.765 INFO:teuthology.orchestra.run.smithi112.stdout:4439 write 0x7ed46ee thru 0x7ee21b9 (0xdacc bytes) 2024-09-04T21:54:59.766 INFO:teuthology.orchestra.run.smithi112.stdout:4440 writesame 0xd957a32 thru 0xd959f6c data_size 0x9(0x253b bytes) 2024-09-04T21:54:59.766 INFO:teuthology.orchestra.run.smithi112.stdout:4441 trunc from 0xdfe3b05 to 0x1cef20d 2024-09-04T21:54:59.766 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:54:59.761+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:54:59.766 INFO:teuthology.orchestra.run.smithi112.stdout:4443 writesame 0x1d6a8ae thru 0x1d7a855 data_size 0x6c(0xffa8 bytes) 2024-09-04T21:54:59.770 INFO:teuthology.orchestra.run.smithi112.stdout:4447 punch from 0x17349b8 to 0x1736ba6, (0x21ee bytes) 2024-09-04T21:54:59.812 INFO:teuthology.orchestra.run.smithi112.stdout:4448 write 0xd7be6fb thru 0xd7cb4c0 (0xcdc6 bytes) 2024-09-04T21:54:59.816 INFO:teuthology.orchestra.run.smithi112.stdout:4449 read 0x73d59ce thru 0x73d7c93 (0x22c6 bytes) 2024-09-04T21:54:59.817 INFO:teuthology.orchestra.run.smithi112.stdout:4451 read 0xb3acc77 thru 0xb3b7cb1 (0xb03b bytes) 2024-09-04T21:54:59.818 INFO:teuthology.orchestra.run.smithi112.stdout:4452 punch from 0x7e68c9 to 0x7f2c49, (0xc380 bytes) 2024-09-04T21:54:59.818 INFO:teuthology.orchestra.run.smithi112.stdout:4453 punch from 0xd32ddfa to 0xd33253f, (0x4745 bytes) 2024-09-04T21:54:59.818 INFO:teuthology.orchestra.run.smithi112.stdout:4454 read 0x744ab2b thru 0x744cd85 (0x225b bytes) 2024-09-04T21:54:59.819 INFO:teuthology.orchestra.run.smithi112.stdout:4455 punch from 0xd4a0f95 to 0xd4ad3a4, (0xc40f bytes) 2024-09-04T21:54:59.819 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:54:59.819 INFO:teuthology.orchestra.run.smithi112.stdout:4457 write 0x5a270d3 thru 0x5a2fbde (0x8b0c bytes) 2024-09-04T21:54:59.823 INFO:teuthology.orchestra.run.smithi112.stdout:4458 writesame 0x9e9d253 thru 0x9ea6f29 data_size 0x1(0x9cd7 bytes) 2024-09-04T21:54:59.865 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:54:59.861+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:54:59.871 INFO:teuthology.orchestra.run.smithi112.stdout:4459 trunc from 0xd7cb4c1 to 0xda917eb 2024-09-04T21:54:59.885 INFO:teuthology.orchestra.run.smithi112.stdout:4460 clone 24 order 18 su 1024 sc 5 2024-09-04T21:54:59.891 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:54:59.891 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:54:59.891 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:54:59.891 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:54:59.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:54:59.965 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:54:59.961+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:00.066 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:00.061+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:00.075 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:00.075 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:00.075 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:00.075 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:00.075 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:00.090 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:00.166 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:00.167 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:00.165+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.710311277999999, 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:00.214 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:00.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365534891999999, 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.219 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365506287000001, 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.36548625, 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365467027999999, 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365448237000001, 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365416294999999, 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365387766, 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365368155000001, 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365349862, 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365327047999999, 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365307911, 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365287244999999, 2024-09-04T21:55:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365256468, 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:00.278 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:00.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.3652371, 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.283 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:00.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365213141, 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.288 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:00.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.292 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365165892, 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:00.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:00.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:00.297 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365140214, 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:00.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:00.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365106188, 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.302 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:00.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.365081786999999, 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:00.311 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:00.313 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:00.313 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:00.314 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:00.314 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:00.313+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:00.321 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:00.322 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:00.338 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:00.338 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:00.354 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:00.355 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:00.373 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:00.373 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:00.391 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:00.412 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:00.412 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:00.413 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:00.409+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:00.431 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:55:00.434 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:00.435 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:00.449 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:00.449 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:00.464 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:00.522 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone23 from 0xda917eb (overlap 0x3a5d6c) to 0x2eb123 2024-09-04T21:55:00.555 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:00.553+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:00.599 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-09-04T21:55:00.614 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:00.609+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:00.639 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:00.639 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:00.653 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:00.739 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:00.737+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:00.815 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:00.813+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:00.820 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:00.820 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:00.833 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:00.917 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:00.917+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:01.000 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:01.000 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:01.014 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:01.017 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:01.017+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:01.023 INFO:teuthology.orchestra.run.smithi112.stdout:4462 punch from 0xd4b6c01 to 0xd4b7433, (0x832 bytes) 2024-09-04T21:55:01.029 INFO:teuthology.orchestra.run.smithi112.stdout:4463 read 0x4ec0f84 thru 0x4ed0ed1 (0xff4e bytes) 2024-09-04T21:55:01.041 INFO:teuthology.orchestra.run.smithi112.stdout:4467 read 0xb616938 thru 0xb61e4e5 (0x7bae bytes) 2024-09-04T21:55:01.045 INFO:teuthology.orchestra.run.smithi112.stdout:4468 write 0xa919785 thru 0xa91c0c6 (0x2942 bytes) 2024-09-04T21:55:01.118 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:01.113+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:01.141 INFO:teuthology.orchestra.run.smithi112.stdout:4469 writesame 0xd88b99a thru 0xd89803a data_size 0x1(0xc6a1 bytes) 2024-09-04T21:55:01.192 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:01.192 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:01.208 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:01.226 INFO:teuthology.orchestra.run.smithi112.stdout:4470 write 0xbe4bf47 thru 0xbe4df70 (0x202a bytes) 2024-09-04T21:55:01.255 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:01.253+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:01.315 INFO:teuthology.orchestra.run.smithi112.stdout:4471 write 0xbc0890e thru 0xbc17003 (0xe6f6 bytes) 2024-09-04T21:55:01.320 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:01.317+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:01.394 INFO:teuthology.orchestra.run.smithi112.stdout:4472 writesame 0xb324cea thru 0xb331a6e data_size 0x1(0xcd85 bytes) 2024-09-04T21:55:01.417 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:01.417 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:01.420 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:01.417+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:01.434 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:55:01.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:01.514 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:01.515 INFO:teuthology.orchestra.run.smithi112.stdout:4474 write 0xa67fcfb thru 0xa68c616 (0xc91c bytes) 2024-09-04T21:55:01.521 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:01.521+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:01.622 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:01.621+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:01.630 INFO:teuthology.orchestra.run.smithi112.stdout:4475 write 0x228fd4e thru 0x2299b26 (0x9dd9 bytes) 2024-09-04T21:55:01.667 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:01.668 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:01.689 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:01.699 INFO:teuthology.orchestra.run.smithi112.stdout:4476 punch from 0x271dca7 to 0x272d5f3, (0xf94c bytes) 2024-09-04T21:55:01.699 INFO:teuthology.orchestra.run.smithi112.stdout:4477 writesame 0x4159acf thru 0x4167ee4 data_size 0x5(0xe416 bytes) 2024-09-04T21:55:01.722 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:01.721+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:01.783 INFO:teuthology.orchestra.run.smithi112.stdout:4478 read 0x45be413 thru 0x45c8cee (0xa8dc bytes) 2024-09-04T21:55:01.787 INFO:teuthology.orchestra.run.smithi112.stdout:4479 write 0xba2faf3 thru 0xba36622 (0x6b30 bytes) 2024-09-04T21:55:01.824 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:01.821+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:01.850 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:01.850 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:01.866 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:01.897 INFO:teuthology.orchestra.run.smithi112.stdout:4481 trunc from 0xda917eb to 0xe46662c 2024-09-04T21:55:01.907 INFO:teuthology.orchestra.run.smithi112.stdout:4484 punch from 0x455e5a to 0x464d17, (0xeebd bytes) 2024-09-04T21:55:01.907 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:01.907 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:01.907 INFO:teuthology.orchestra.run.smithi112.stdout:4488 punch from 0xaa7512 to 0xab6b01, (0xf5ef bytes) 2024-09-04T21:55:01.907 INFO:teuthology.orchestra.run.smithi112.stdout:4489 writesame 0xe2d0f7e thru 0xe2d9ced data_size 0x7c(0x8d70 bytes) 2024-09-04T21:55:01.925 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:01.925+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:01.926 INFO:teuthology.orchestra.run.smithi112.stdout:4491 writesame 0x3a81002 thru 0x3a8cc76 data_size 0x1(0xbc75 bytes) 2024-09-04T21:55:02.005 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:02.005 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:02.022 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:02.034 INFO:teuthology.orchestra.run.smithi112.stdout:4492 write 0x2aca670 thru 0x2ad8eb0 (0xe841 bytes) 2024-09-04T21:55:02.067 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:02.065+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:02.127 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:02.125+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:02.153 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:02.153 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:02.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:02.188 INFO:teuthology.orchestra.run.smithi112.stdout:4494 writesame 0x39e9f1c thru 0x39ed0e4 data_size 0x1(0x31c9 bytes) 2024-09-04T21:55:02.220 INFO:teuthology.orchestra.run.smithi112.stdout:4495 trunc from 0xe46662c to 0x2d4772a 2024-09-04T21:55:02.228 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:02.225+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:02.328 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:02.325+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:02.360 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:02.360 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:02.375 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:02.418 INFO:teuthology.orchestra.run.smithi112.stdout:4496 trunc from 0x2d4772a to 0xed37e7c 2024-09-04T21:55:02.430 INFO:teuthology.orchestra.run.smithi112.stdout:4497 read 0x5dc2f80 thru 0x5dc372b (0x7ac bytes) 2024-09-04T21:55:02.430 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:02.435 INFO:teuthology.orchestra.run.smithi112.stdout:4500 trunc from 0xed37e7c to 0xc83a124 2024-09-04T21:55:02.448 INFO:teuthology.orchestra.run.smithi112.stdout:4502 writesame 0xb4e10a0 thru 0xb4eef93 data_size 0x26(0xdef4 bytes) 2024-09-04T21:55:02.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:02.463 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:02.461+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:02.463 INFO:teuthology.orchestra.run.smithi112.stdout:4503 read 0x35d12f1 thru 0x35d324f (0x1f5f bytes) 2024-09-04T21:55:02.463 INFO:teuthology.orchestra.run.smithi112.stdout:4505 writesame 0x106c716 thru 0x107b4ba data_size 0x3(0xeda5 bytes) 2024-09-04T21:55:02.530 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:02.529+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:02.555 INFO:teuthology.orchestra.run.smithi112.stdout:4506 read 0x4ffebe5 thru 0x500be3b (0xd257 bytes) 2024-09-04T21:55:02.555 INFO:teuthology.orchestra.run.smithi112.stdout:4507 write 0xa4b0380 thru 0xa4b0fc2 (0xc43 bytes) 2024-09-04T21:55:02.561 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:02.562 INFO:teuthology.orchestra.run.smithi112.stdout:4509 read 0x70d4c00 thru 0x70e36e6 (0xeae7 bytes) 2024-09-04T21:55:02.562 INFO:teuthology.orchestra.run.smithi112.stdout:4512 write 0x5542f63 thru 0x554758f (0x462d bytes) 2024-09-04T21:55:02.564 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:02.564 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:02.569 INFO:teuthology.orchestra.run.smithi112.stdout:4513 read 0x909aa78 thru 0x90a4162 (0x96eb bytes) 2024-09-04T21:55:02.570 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:02.576 INFO:teuthology.orchestra.run.smithi112.stdout:4515 write 0xcf58b04 thru 0xcf64cda (0xc1d7 bytes) 2024-09-04T21:55:02.579 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:02.601 INFO:teuthology.orchestra.run.smithi112.stdout:4516 trunc from 0xcf64cdb to 0x182be8e 2024-09-04T21:55:02.626 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:02.626 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:02.631 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:02.629+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:02.652 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:02.652 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:02.678 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:02.678 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:02.683 INFO:teuthology.orchestra.run.smithi112.stdout:4517 read 0x1636488 thru 0x1637618 (0x1191 bytes) 2024-09-04T21:55:02.684 INFO:teuthology.orchestra.run.smithi112.stdout:4520 writesame 0x12a1c96 thru 0x12aecd3 data_size 0xf(0xd03e bytes) 2024-09-04T21:55:02.703 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:02.704 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:02.721 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:02.721 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:02.721 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:02.721 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:02.723 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:02.724 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:02.732 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:02.729+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:02.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:02.744 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:02.744 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:02.762 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:02.762 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:02.778 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:02.778 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:02.781 INFO:teuthology.orchestra.run.smithi112.stdout:4521 writesame 0xaaae28 thru 0xaab524 data_size 0x1(0x6fd bytes) 2024-09-04T21:55:02.845 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:02.845+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:02.849 INFO:teuthology.orchestra.run.smithi112.stdout:4522 writesame 0x74690ef thru 0x746f1a5 data_size 0x9(0x60b7 bytes) 2024-09-04T21:55:02.849 INFO:teuthology.orchestra.run.smithi112.stdout:4523 read 0x6c6591a thru 0x6c68904 (0x2feb bytes) 2024-09-04T21:55:02.849 INFO:teuthology.orchestra.run.smithi112.stdout:4528 write 0x1f18a25 thru 0x1f251b0 (0xc78c bytes) 2024-09-04T21:55:02.864 INFO:teuthology.orchestra.run.smithi112.stdout:4529 trunc from 0x746f1a6 to 0x27260cd 2024-09-04T21:55:02.870 INFO:teuthology.orchestra.run.smithi112.stdout:4530 read 0xdf636a thru 0xe00ef1 (0xab88 bytes) 2024-09-04T21:55:02.879 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:02.879 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:02.879 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:02.879 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:02.879 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:02.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:02.904 INFO:teuthology.orchestra.run.smithi112.stdout:4531 writesame 0xce06a79 thru 0xce0b59e data_size 0x1(0x4b26 bytes) 2024-09-04T21:55:02.913 INFO:teuthology.orchestra.run.smithi112.stdout:4534 write 0xd7e01a4 thru 0xd7eb8c4 (0xb721 bytes) 2024-09-04T21:55:02.926 INFO:teuthology.orchestra.run.smithi112.stdout:4535 trunc from 0xd7eb8c5 to 0xe48fbcf 2024-09-04T21:55:02.933 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:02.933 INFO:teuthology.orchestra.run.smithi112.stdout:4537 write 0x868248 thru 0x86e931 (0x66ea bytes) 2024-09-04T21:55:02.934 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:02.929+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:03.025 INFO:teuthology.orchestra.run.smithi112.stdout:4539 punch from 0x2b7d53e to 0x2b85ae8, (0x85aa bytes) 2024-09-04T21:55:03.034 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:03.029+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:03.035 INFO:teuthology.orchestra.run.smithi112.stdout:4540 trunc from 0xe48fbcf to 0x7dff7c0 2024-09-04T21:55:03.041 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366175128999998, 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.042 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366148344000003, 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366119411, 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366095764000001, 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366075983000002, 2024-09-04T21:55:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366056587999999, 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366037315, 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.366018093000001, 2024-09-04T21:55:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:03.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365999256000002, 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:03.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:03.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365979852999999, 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.085 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:03.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.089 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365958796999998, 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:03.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365926619, 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.094 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:03.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365900398000001, 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:03.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:03.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365872189000001, 2024-09-04T21:55:03.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:03.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:03.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365852148999998, 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.108 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:03.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365832767000001, 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:03.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.114 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:03.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365809179000003, 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365788862000002, 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365769309999997, 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:03.128 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:03.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.365749551, 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.137 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:03.138 INFO:teuthology.orchestra.run.smithi112.stdout:4541 write 0x1585acb thru 0x158a66d (0x4ba3 bytes) 2024-09-04T21:55:03.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:4542 read 0x2a5af85 thru 0x2a62888 (0x7904 bytes) 2024-09-04T21:55:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:4543 writesame 0x780394d thru 0x78109ae data_size 0x33(0xd062 bytes) 2024-09-04T21:55:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:4544 write 0x5fe5200 thru 0x5fe9fbe (0x4dbf bytes) 2024-09-04T21:55:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:4546 write 0x86d3653 thru 0x86dcc38 (0x95e6 bytes) 2024-09-04T21:55:03.141 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:03.137+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:03.150 INFO:teuthology.orchestra.run.smithi112.stdout:4547 punch from 0x48e26cd to 0x48e82b2, (0x5be5 bytes) 2024-09-04T21:55:03.150 INFO:teuthology.orchestra.run.smithi112.stdout:4548 punch from 0x7c86590 to 0x7c93ec6, (0xd936 bytes) 2024-09-04T21:55:03.150 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:03.150 INFO:teuthology.orchestra.run.smithi112.stdout:4552 punch from 0x7ea5e06 to 0x7eb54ad, (0xf6a7 bytes) 2024-09-04T21:55:03.150 INFO:teuthology.orchestra.run.smithi112.stdout:4554 write 0x5ba9d02 thru 0x5baea30 (0x4d2f bytes) 2024-09-04T21:55:03.163 INFO:teuthology.orchestra.run.smithi112.stdout:4555 write 0x5893c4e thru 0x589a038 (0x63eb bytes) 2024-09-04T21:55:03.171 INFO:teuthology.orchestra.run.smithi112.stdout:4556 clone 25 order 25 su 16777216 sc 4 2024-09-04T21:55:03.239 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:03.237+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:03.268 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:03.268 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:03.268 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:03.269 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:03.282 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:03.340 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:03.337+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:03.441 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:03.441+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:03.454 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:03.454 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:03.454 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:03.454 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:03.454 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:03.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:55:03.469 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:03.542 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:03.537+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:03.575 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone24 from 0x86dcc39 (overlap 0x182be8e) to 0x10eb896 2024-09-04T21:55:03.643 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:03.641+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.802913875000002, 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:55:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:55:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:55:03.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.662 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.676866611999998, 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.480312974, 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:55:03.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:55:03.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:03.674 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.452923476000002, 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:55:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:55:03.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.429659004999998, 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.679 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:55:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.401901215000002, 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:55:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:55:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.401785652999997, 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:55:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:55:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.101610868000002, 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:55:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:55:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.942478315999999, 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:55:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.942252258000003, 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:55:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:55:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.465039189999999, 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:55:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:55:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:55:03.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.235651079, 2024-09-04T21:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:55:03.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:55:03.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:55:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.937460438000002, 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:55:03.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:55:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.937347582999998, 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:55:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:55:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.927686614999999, 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.743 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:55:03.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:55:03.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:55:03.747 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.907836889999999, 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:55:03.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:55:03.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:03.752 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.757608204, 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:03.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.091281299000002, 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:55:03.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:55:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.088142113000004, 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:55:03.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.069831123, 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:55:03.769 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:03.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:55:03.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:03.774 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:03.774 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:03.775 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:03.776 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:03.773+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:03.876 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:03.873+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:03.941 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:03.941 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:03.957 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:04.001 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:04.001+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:04.056 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-09-04T21:55:04.077 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:04.077+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:04.077 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:04.077 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:04.078 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:04.078 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:04.078 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:04.089 INFO:teuthology.orchestra.run.smithi112.stdout:4557 write 0xa31d7e8 thru 0xa31e1f5 (0xa0e bytes) 2024-09-04T21:55:04.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:4558 punch from 0x1f63066 to 0x1f6f5c7, (0xc561 bytes) 2024-09-04T21:55:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:4559 read 0x4a2be56 thru 0x4a362fa (0xa4a5 bytes) 2024-09-04T21:55:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:4560 read 0x2d3896f thru 0x2d3b622 (0x2cb4 bytes) 2024-09-04T21:55:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:4561 read 0x9c03cd9 thru 0x9c13620 (0xf948 bytes) 2024-09-04T21:55:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:4562 punch from 0x6681e9e to 0x668b9f0, (0x9b52 bytes) 2024-09-04T21:55:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:4564 read 0x17808ec thru 0x178210b (0x1820 bytes) 2024-09-04T21:55:04.101 INFO:teuthology.orchestra.run.smithi112.stdout:4565 punch from 0x698ed6e to 0x6999bd1, (0xae63 bytes) 2024-09-04T21:55:04.102 INFO:teuthology.orchestra.run.smithi112.stdout:4566 trunc from 0xa31e1f6 to 0x9d72cf1 2024-09-04T21:55:04.122 INFO:teuthology.orchestra.run.smithi112.stdout:4568 punch from 0xe09e1f to 0xe12fbe, (0x919f bytes) 2024-09-04T21:55:04.122 INFO:teuthology.orchestra.run.smithi112.stdout:4570 punch from 0x555f5f2 to 0x556bb9a, (0xc5a8 bytes) 2024-09-04T21:55:04.122 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:04.122 INFO:teuthology.orchestra.run.smithi112.stdout:4574 write 0x4fd9f2b thru 0x4fdc484 (0x255a bytes) 2024-09-04T21:55:04.177 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:04.173+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:04.231 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32654 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 4800512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.640044461999999, 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027786758000000002, 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.232 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32654 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:04.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588400+0000", 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3399999999999999e-07 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588403+0000", 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7249999999999999e-06 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588414+0000", 2024-09-04T21:55:04.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0559e-05 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588421+0000", 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5109999999999997e-06 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590927+0000", 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025057339999999999 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590933+0000", 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5820000000000001e-06 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610352+0000", 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.019419174000000001 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610442+0000", 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0125999999999997e-05 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.616186+0000", 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057437130000000001 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32670 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5062656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639430666000003, 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024152731, 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32670 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4199999999999995e-07 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589017+0000", 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.224e-06 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589031+0000", 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4747e-05 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589039+0000", 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591456+0000", 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024168549999999999 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591462+0000", 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4979999999999997e-06 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613066+0000", 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0216037 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613120+0000", 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4209999999999998e-05 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613165+0000", 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5763999999999998e-05 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32671 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5079040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639391258000003, 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024282133000000001, 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32671 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:04.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589053+0000", 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589057+0000", 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589074+0000", 2024-09-04T21:55:04.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7201999999999999e-05 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589083+0000", 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5739999999999996e-06 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591487+0000", 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024049150000000001 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591493+0000", 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-06 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613189+0000", 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021695945000000001 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613260+0000", 2024-09-04T21:55:04.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0905e-05 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613334+0000", 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4425999999999995e-05 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32672 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5095424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:04.247 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639348654999999, 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024400369000000002, 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32672 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1299999999999999e-07 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589099+0000", 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.591e-06 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589115+0000", 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5591999999999999e-05 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589128+0000", 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3210000000000001e-05 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591520+0000", 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023920529999999999 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591526+0000", 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613358+0000", 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021832949000000001 2024-09-04T21:55:04.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613427+0000", 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8955999999999995e-05 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613495+0000", 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7694999999999995e-05 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32673 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639304525, 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024527538000000002, 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32673 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589140+0000", 2024-09-04T21:55:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589143+0000", 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.331e-06 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589153+0000", 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7270000000000002e-06 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589161+0000", 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8149999999999995e-06 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591553+0000", 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023916469999999998 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591558+0000", 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1649999999999997e-06 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613520+0000", 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021961729999999999 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613591+0000", 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0735000000000006e-05 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613666+0000", 2024-09-04T21:55:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5754999999999996e-05 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32674 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5128192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639271033, 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024653952999999999, 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32674 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589173+0000", 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000005e-07 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589186+0000", 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2972999999999999e-05 2024-09-04T21:55:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589204+0000", 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7666999999999999e-05 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589212+0000", 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3999999999999992e-06 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591589+0000", 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023771370000000001 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591594+0000", 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3149999999999999e-06 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613690+0000", 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022095789000000001 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613759+0000", 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9142999999999999e-05 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613826+0000", 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6958999999999994e-05 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.263 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32675 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5144576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639218614000001, 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024775136, 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32675 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589226+0000", 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0500000000000003e-07 2024-09-04T21:55:04.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589229+0000", 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999998e-06 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589239+0000", 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589247+0000", 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0430000000000008e-06 2024-09-04T21:55:04.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591622+0000", 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023745099999999998 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591627+0000", 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613851+0000", 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022223613 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613923+0000", 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2203e-05 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614000+0000", 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.268 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32676 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5160960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639185376999997, 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024914054000000001, 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32676 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589259+0000", 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589276+0000", 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7686e-05 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589288+0000", 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1234e-05 2024-09-04T21:55:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589296+0000", 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5769999999999994e-06 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591654+0000", 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023581769999999999 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591660+0000", 2024-09-04T21:55:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3109999999999996e-06 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614025+0000", 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022365051 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614099+0000", 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3881000000000006e-05 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614172+0000", 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3498999999999995e-05 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32677 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5177344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639127346000002, 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025050084, 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32677 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589317+0000", 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5300000000000003e-07 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589329+0000", 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1951e-05 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589346+0000", 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7108e-05 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589354+0000", 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2630000000000003e-06 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591674+0000", 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023197980000000001 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591677+0000", 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5909999999999998e-06 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614205+0000", 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022528256999999999 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614287+0000", 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1886999999999997e-05 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614366+0000", 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9475999999999996e-05 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32678 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5193728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639075153999997, 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025172158, 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.279 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32678 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589369+0000", 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8400000000000004e-07 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589373+0000", 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589391+0000", 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8077e-05 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589400+0000", 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0790000000000007e-06 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591689+0000", 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002288815 2024-09-04T21:55:04.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591692+0000", 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.666e-06 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614392+0000", 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022700287 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614462+0000", 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0468000000000004e-05 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614540+0000", 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8104999999999998e-05 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.284 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32679 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5210112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.639031221000003, 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025302998, 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32679 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589413+0000", 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0100000000000004e-07 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589417+0000", 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7689999999999998e-06 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589435+0000", 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8185e-05 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589444+0000", 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4029999999999996e-06 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591704+0000", 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022601729999999999 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591708+0000", 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.067e-06 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614571+0000", 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022863270000000002 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614640+0000", 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909e-05 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614715+0000", 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5520999999999997e-05 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32680 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5226496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.638985611000003, 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025450210000000001, 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.290 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32680 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589459+0000", 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589462+0000", 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7780000000000001e-06 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589479+0000", 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6651e-05 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589488+0000", 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6309999999999992e-06 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591737+0000", 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002249301 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591742+0000", 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4790000000000004e-06 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614747+0000", 2024-09-04T21:55:04.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.023004539000000001 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614825+0000", 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8566000000000006e-05 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614908+0000", 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2558999999999993e-05 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.295 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32929 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5120000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629369746999998, 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024146612000000001, 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.296 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32929 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599077+0000", 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.373e-06 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599083+0000", 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5060000000000001e-06 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599088+0000", 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0690000000000001e-06 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601461+0000", 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023728299999999998 2024-09-04T21:55:04.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601465+0000", 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5129999999999998e-06 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623133+0000", 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021667782 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623178+0000", 2024-09-04T21:55:04.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4950000000000002e-05 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623220+0000", 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2076999999999997e-05 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32930 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5136384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:04.301 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629348022999999, 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024238576000000001, 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32930 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599096+0000", 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599098+0000", 2024-09-04T21:55:04.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2979999999999999e-06 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599105+0000", 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5479999999999998e-06 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599110+0000", 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1109999999999999e-06 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601491+0000", 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023813800000000002 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601496+0000", 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623247+0000", 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021751177 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623291+0000", 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4184999999999998e-05 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623334+0000", 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2970999999999999e-05 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.306 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32931 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5152768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629325756999997, 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024330243000000001, 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32931 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.61e-07 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599121+0000", 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.683e-06 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599129+0000", 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6359999999999994e-06 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599134+0000", 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5049999999999998e-06 2024-09-04T21:55:04.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601519+0000", 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002384546 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601521+0000", 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2919999999999998e-06 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623360+0000", 2024-09-04T21:55:04.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021839164000000001 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623405+0000", 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4243999999999999e-05 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623448+0000", 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3411999999999997e-05 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32932 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5169152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629300723, 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024418507999999998, 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32932 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.312 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599144+0000", 2024-09-04T21:55:04.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4799999999999997e-07 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599153+0000", 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9930000000000001e-06 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599161+0000", 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0900000000000005e-06 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599167+0000", 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601543+0000", 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023758540000000002 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601547+0000", 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1019999999999999e-06 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623474+0000", 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021927782 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623519+0000", 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4263e-05 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623561+0000", 2024-09-04T21:55:04.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2483e-05 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32933 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5185536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629266628000003, 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024496766999999999, 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.317 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32933 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:04.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-07 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599180+0000", 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599188+0000", 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599203+0000", 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5099999999999999e-05 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601573+0000", 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002369566 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601577+0000", 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5090000000000004e-06 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623587+0000", 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022009635999999999 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623631+0000", 2024-09-04T21:55:04.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4605999999999999e-05 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623674+0000", 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2274000000000002e-05 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32934 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5201920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629230456000002, 2024-09-04T21:55:04.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024572765999999999, 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32934 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599214+0000", 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599217+0000", 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.095e-06 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599224+0000", 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6760000000000007e-06 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599230+0000", 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8039999999999998e-06 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601603+0000", 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023727090000000002 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601608+0000", 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-06 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623700+0000", 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022092219999999999 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623744+0000", 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4013e-05 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623786+0000", 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1749000000000003e-05 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32935 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5218304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629204172000001, 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024658065999999999, 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32935 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599240+0000", 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3500000000000002e-07 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599249+0000", 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599258+0000", 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7069999999999998e-06 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599264+0000", 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4790000000000001e-06 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601634+0000", 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023694020000000001 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601638+0000", 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623812+0000", 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022173986 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623856+0000", 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3696000000000003e-05 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623897+0000", 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1530999999999999e-05 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.332 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32936 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5234688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.629168821, 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024740649, 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32936 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2099999999999997e-07 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599278+0000", 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8870000000000001e-06 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599285+0000", 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4449999999999999e-06 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599291+0000", 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601668+0000", 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023769379999999999 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601673+0000", 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8210000000000001e-06 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623927+0000", 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022253536000000001 2024-09-04T21:55:04.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623972+0000", 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5216999999999998e-05 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.624015+0000", 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3519000000000001e-05 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.338 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:04.341 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:04.337+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:04.341 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:04.441 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:04.437+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:04.445 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:55:04.530 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:96101 3.5 3:a7bc27db:::rbd_data.1188b9e2364f.0000000000000005:1a [read 1036288~1024 out=1024b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:04.531118+0000", 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.000399038, 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040536400000000002, 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 96101 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531118+0000", 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531118+0000", 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531118+0000", 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1099999999999995e-07 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531121+0000", 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9119999999999998e-06 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531129+0000", 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3919999999999997e-06 2024-09-04T21:55:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531135+0000", 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9109999999999998e-06 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531408+0000", 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027381600000000002 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531448+0000", 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9888999999999997e-05 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531523+0000", 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4833000000000007e-05 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:96103 3.5 3:a7bc27db:::rbd_data.1188b9e2364f.0000000000000005:1a [read 1041408~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:04.531238+0000", 2024-09-04T21:55:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.00027896699999999997, 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00037794, 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 96103 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531238+0000", 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531238+0000", 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531238+0000", 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4799999999999998e-07 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531241+0000", 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531248+0000", 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0190000000000001e-06 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531254+0000", 2024-09-04T21:55:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6149999999999996e-06 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531543+0000", 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028941900000000002 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531586+0000", 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2388999999999997e-05 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:96105 3.5 3.dbe43de5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:04.531304+0000", 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.00021302699999999999, 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000366416, 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 96105 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531304+0000", 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531304+0000", 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531304+0000", 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6100000000000001e-07 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531307+0000", 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8169999999999999e-06 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531313+0000", 2024-09-04T21:55:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.319e-06 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531319+0000", 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-06 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:04.531672+0000", 2024-09-04T21:55:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000353147 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 3 2024-09-04T21:55:04.545 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:04.546 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:04.547 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:04.541+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:04.570 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7] out_osds: [1, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:55:04.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-04T21:55:04.647 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:04.645+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:04.682 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:04.682 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:04.683 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:04.683 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:04.683 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:04.703 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:04.749 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:04.749+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:04.800 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:04.850 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:04.849+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:04.920 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:04.920 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:04.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:04.920 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:04.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.416354165999998, 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:04.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:04.925 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071577779999998, 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:04.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071549175000001, 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.930 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071529137999999, 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.935 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:04.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.939 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071509916, 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:04.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:04.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071491125000001, 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:04.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071459183000002, 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.949 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.950 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:04.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:04.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071430654, 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:04.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:04.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.960 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071411043000001, 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:04.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071392750000001, 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:04.965 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:04.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071369936, 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:04.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:04.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071350799000001, 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:04.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:04.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.979 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071330133, 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:04.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:04.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071299356000001, 2024-09-04T21:55:04.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:04.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071279988000001, 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:04.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071256029000001, 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.994 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:04.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:04.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071208779999999, 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:04.999 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:05.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:05.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071183101999999, 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:05.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:05.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:05.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:05.008 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071149076000001, 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:05.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:05.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.071124675, 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:05.013 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:05.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:05.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:05.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:05.018 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:05.018 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:05.019 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:05.020 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:05.020 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:05.020 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:05.021 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:05.021 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:05.017+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:05.037 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:05.054 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:05.071 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:05.071 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:05.089 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:05.089 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:05.107 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:05.107 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:05.119 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:05.117+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:05.124 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:05.124 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:05.144 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:05.144 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:05.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:05.243 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:05.241+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:05.309 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:05.309 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:05.322 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:05.322 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:05.321+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:05.422 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:05.421+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:05.461 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.2 to 1 (10000) 2024-09-04T21:55:05.480 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:05.480 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:05.482 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-04T21:55:05.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:05.539 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:05.537+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:05.610 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:05.610 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:05.623 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:05.624 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:05.621+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:05.724 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:05.721+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:05.780 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:05.780 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:05.793 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:05.839 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:05.837+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:05.926 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:05.925+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:05.955 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:05.955 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:05.970 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:06.027 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:06.025+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:06.078 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.078 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.091 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:06.135 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:06.133+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:06.199 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.199 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.213 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:06.255 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:06.253+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:06.329 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:06.329+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:06.370 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.370 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.383 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:06.430 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:06.429+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:06.463 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.3 to 1 (10000) 2024-09-04T21:55:06.475 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:55:06.487 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-04T21:55:06.500 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.500 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.513 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:06.573 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:06.573+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:06.632 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:06.629+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:06.633 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.633 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.647 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:06.733 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:06.729+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:06.768 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.768 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.782 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:06.834 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:06.833+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:06.895 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:06.895 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:06.909 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:06.935 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:06.933+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:07.035 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:07.033+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:07.070 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:07.070 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:07.085 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:07.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:07.159 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.159 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.159 INFO:teuthology.orchestra.run.smithi112.stdout:4580 punch from 0x5897e64 to 0x589de2a, (0x5fc6 bytes) 2024-09-04T21:55:07.159 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:07.157+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:07.165 INFO:teuthology.orchestra.run.smithi112.stdout:4581 trunc from 0x9d72cf1 to 0x546c777 2024-09-04T21:55:07.237 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:07.233+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:07.265 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:07.265 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:07.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:07.319 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:07.319 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:07.338 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:07.333+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:07.338 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:07.339 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:07.355 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:07.355 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:07.371 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:07.372 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:07.387 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:07.388 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:07.404 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:07.405 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:07.421 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:07.421 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:07.434 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:07.434 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:07.434 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:07.434 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:07.438 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:07.437+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:07.441 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:07.448 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:55:07.480 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.4 to 1 (10000) 2024-09-04T21:55:07.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-04T21:55:07.539 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:07.537+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:07.594 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:07.594 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:07.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:07.619 INFO:teuthology.orchestra.run.smithi112.stdout:4583 write 0x308c4db thru 0x3096019 (0x9b3f bytes) 2024-09-04T21:55:07.623 INFO:teuthology.orchestra.run.smithi112.stdout:4584 read 0x12a7996 thru 0x12b25f9 (0xac64 bytes) 2024-09-04T21:55:07.633 INFO:teuthology.orchestra.run.smithi112.stdout:4585 trunc from 0x546c777 to 0xc9ae60f 2024-09-04T21:55:07.640 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:07.637+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:07.652 INFO:teuthology.orchestra.run.smithi112.stdout:4586 trunc from 0xc9ae60f to 0x29237e5 2024-09-04T21:55:07.687 INFO:teuthology.orchestra.run.smithi112.stdout:4587 writesame 0x7faeb57 thru 0x7fb7ae0 data_size 0x13(0x8f8a bytes) 2024-09-04T21:55:07.687 INFO:teuthology.orchestra.run.smithi112.stdout:4590 punch from 0x21c0c34 to 0x21ccb64, (0xbf30 bytes) 2024-09-04T21:55:07.701 INFO:teuthology.orchestra.run.smithi112.stdout:4591 write 0xc7488df thru 0xc74e2c8 (0x59ea bytes) 2024-09-04T21:55:07.708 INFO:teuthology.orchestra.run.smithi112.stdout:4592 read 0x905cf5a thru 0x905dbc4 (0xc6b bytes) 2024-09-04T21:55:07.708 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.708 INFO:teuthology.orchestra.run.smithi112.stdout:4594 punch from 0x63f0556 to 0x63fb765, (0xb20f bytes) 2024-09-04T21:55:07.708 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.708 INFO:teuthology.orchestra.run.smithi112.stdout:4597 write 0x193c46d thru 0x193f31f (0x2eb3 bytes) 2024-09-04T21:55:07.711 INFO:teuthology.orchestra.run.smithi112.stdout:4600 read 0x6b33b37 thru 0x6b43228 (0xf6f2 bytes) 2024-09-04T21:55:07.711 INFO:teuthology.orchestra.run.smithi112.stdout:4602 write 0x5642b2e thru 0x565175d (0xec30 bytes) 2024-09-04T21:55:07.716 INFO:teuthology.orchestra.run.smithi112.stdout:4603 read 0xb8443c8 thru 0xb84c94a (0x8583 bytes) 2024-09-04T21:55:07.716 INFO:teuthology.orchestra.run.smithi112.stdout:4604 read 0x58dbaf0 thru 0x58e1f24 (0x6435 bytes) 2024-09-04T21:55:07.717 INFO:teuthology.orchestra.run.smithi112.stdout:4605 punch from 0xa6577ed to 0xa658385, (0xb98 bytes) 2024-09-04T21:55:07.728 INFO:teuthology.orchestra.run.smithi112.stdout:4606 writesame 0xeab67cc thru 0xeabc165 data_size 0x3(0x599a bytes) 2024-09-04T21:55:07.729 INFO:teuthology.orchestra.run.smithi112.stdout:4607 read 0x2c7dec2 thru 0x2c897b3 (0xb8f2 bytes) 2024-09-04T21:55:07.729 INFO:teuthology.orchestra.run.smithi112.stdout:4608 write 0xdbad8ab thru 0xdbb97d6 (0xbf2c bytes) 2024-09-04T21:55:07.740 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:07.741+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:07.748 INFO:teuthology.orchestra.run.smithi112.stdout:4609 trunc from 0xeabc166 to 0x6e756c7 2024-09-04T21:55:07.750 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.897170040999995, 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.751 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:55:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:55:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.771122778000006, 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:55:07.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.574569139999994, 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.763 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:55:07.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:55:07.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:55:07.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.547179642000003, 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:55:07.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.523915170999999, 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:55:07.773 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:55:07.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.496157381000003, 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.779 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:55:07.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:55:07.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:55:07.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.496041818999998, 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.785 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:55:07.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:55:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.195867034000003, 2024-09-04T21:55:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:55:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:55:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 61.036734482, 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:55:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:55:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 61.036508423999997, 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:55:07.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:55:07.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:55:07.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:55:07.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:55:07.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:55:07.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.559295356, 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.813 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:55:07.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.329907245000001, 2024-09-04T21:55:07.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:07.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:55:07.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:55:07.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.031716604000003, 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:55:07.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:55:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:55:07.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:07.832 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.031603748999999, 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:55:07.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:55:07.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.021942781, 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:55:07.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:55:07.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:55:07.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 60.002093056, 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.844 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:55:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.851864370000001, 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:55:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:55:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.185537465000003, 2024-09-04T21:55:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:55:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:55:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.182398278999997, 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:55:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.164087289000001, 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:55:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:07.871 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:07.873 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:07.874 INFO:teuthology.orchestra.run.smithi112.stdout:4610 write 0x833698 thru 0x83ccee (0x9657 bytes) 2024-09-04T21:55:07.874 INFO:teuthology.orchestra.run.smithi112.stdout:4611 trunc from 0x6e756c7 to 0xb624707 2024-09-04T21:55:07.874 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.874 INFO:teuthology.orchestra.run.smithi112.stdout:4614 read 0x5bc08aa thru 0x5bc6915 (0x606c bytes) 2024-09-04T21:55:07.874 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:4616 punch from 0x446d39a to 0x4473421, (0x6087 bytes) 2024-09-04T21:55:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:4617 writesame 0xb39375e thru 0xb3994f2 data_size 0x1(0x5d95 bytes) 2024-09-04T21:55:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:4618 write 0x978243b thru 0x978efd5 (0xcb9b bytes) 2024-09-04T21:55:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:4620 trunc from 0xb624707 to 0x530cdc5 2024-09-04T21:55:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:4624 trunc from 0x530cdc5 to 0x3107b68 2024-09-04T21:55:07.875 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:07.869+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:07.876 INFO:teuthology.orchestra.run.smithi112.stdout:4626 read 0x11ac503 thru 0x11b723b (0xad39 bytes) 2024-09-04T21:55:07.876 INFO:teuthology.orchestra.run.smithi112.stdout:4627 read 0x2afce77 thru 0x2b0b300 (0xe48a bytes) 2024-09-04T21:55:07.878 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.878 INFO:teuthology.orchestra.run.smithi112.stdout:4629 read 0x1934499 thru 0x193a9b3 (0x651b bytes) 2024-09-04T21:55:07.878 INFO:teuthology.orchestra.run.smithi112.stdout:4630 punch from 0x23be5e8 to 0x23c4c13, (0x662b bytes) 2024-09-04T21:55:07.880 INFO:teuthology.orchestra.run.smithi112.stdout:4631 trunc from 0x3107b68 to 0x161a40d 2024-09-04T21:55:07.899 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:07.899 INFO:teuthology.orchestra.run.smithi112.stdout:4633 read 0x160e125 thru 0x161a40c (0xc2e8 bytes) 2024-09-04T21:55:07.900 INFO:teuthology.orchestra.run.smithi112.stdout:4634 punch from 0xfb2f3e to 0xfb59dd, (0x2a9f bytes) 2024-09-04T21:55:07.918 INFO:teuthology.orchestra.run.smithi112.stdout:4639 write 0x7317d94 thru 0x7324f47 (0xd1b4 bytes) 2024-09-04T21:55:07.926 INFO:teuthology.orchestra.run.smithi112.stdout:4640 trunc from 0x7324f48 to 0x679a3ee 2024-09-04T21:55:07.943 INFO:teuthology.orchestra.run.smithi112.stdout:4641 trunc from 0x679a3ee to 0x38005fb 2024-09-04T21:55:07.957 INFO:teuthology.orchestra.run.smithi112.stdout:4642 punch from 0x1128556 to 0x1136449, (0xdef3 bytes) 2024-09-04T21:55:07.970 INFO:teuthology.orchestra.run.smithi112.stdout:4644 write 0x5761cd3 thru 0x5762a4d (0xd7b bytes) 2024-09-04T21:55:07.974 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:07.973+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:07.974 INFO:teuthology.orchestra.run.smithi112.stdout:4645 write 0x7a96fd thru 0x7b133f (0x7c43 bytes) 2024-09-04T21:55:07.978 INFO:teuthology.orchestra.run.smithi112.stdout:4646 punch from 0x175fbef to 0x17601b4, (0x5c5 bytes) 2024-09-04T21:55:07.978 INFO:teuthology.orchestra.run.smithi112.stdout:4647 punch from 0x3644b39 to 0x364e2e5, (0x97ac bytes) 2024-09-04T21:55:07.994 INFO:teuthology.orchestra.run.smithi112.stdout:4648 write 0x9845b63 thru 0x984aaa7 (0x4f45 bytes) 2024-09-04T21:55:08.002 INFO:teuthology.orchestra.run.smithi112.stdout:4649 clone 26 order 19 su 131072 sc 15 2024-09-04T21:55:08.007 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:08.008 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:08.008 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:08.008 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:08.020 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:08.074 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:08.073+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:08.175 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:08.173+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:08.183 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:08.184 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:08.184 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:08.184 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:08.184 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:08.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:08.285 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:08.285+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:08.359 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32654 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 4800512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.770518864000003, 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027786758000000002, 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.360 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32654 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:08.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588400+0000", 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3399999999999999e-07 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588403+0000", 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7249999999999999e-06 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588414+0000", 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0559e-05 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588421+0000", 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5109999999999997e-06 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590927+0000", 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025057339999999999 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590933+0000", 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5820000000000001e-06 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610352+0000", 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.019419174000000001 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610442+0000", 2024-09-04T21:55:08.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0125999999999997e-05 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.616186+0000", 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057437130000000001 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32670 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5062656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:08.365 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769905068, 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024152731, 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32670 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4199999999999995e-07 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589017+0000", 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.224e-06 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589031+0000", 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4747e-05 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589039+0000", 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-04T21:55:08.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591456+0000", 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024168549999999999 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591462+0000", 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4979999999999997e-06 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613066+0000", 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0216037 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613120+0000", 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4209999999999998e-05 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613165+0000", 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5763999999999998e-05 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32671 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5079040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769865660000001, 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024282133000000001, 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32671 2024-09-04T21:55:08.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589053+0000", 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589057+0000", 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589074+0000", 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7201999999999999e-05 2024-09-04T21:55:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589083+0000", 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5739999999999996e-06 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591487+0000", 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024049150000000001 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591493+0000", 2024-09-04T21:55:08.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-06 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613189+0000", 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021695945000000001 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613260+0000", 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0905e-05 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613334+0000", 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4425999999999995e-05 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32672 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5095424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769823056999996, 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024400369000000002, 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32672 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:08.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1299999999999999e-07 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589099+0000", 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.591e-06 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589115+0000", 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5591999999999999e-05 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589128+0000", 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3210000000000001e-05 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591520+0000", 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023920529999999999 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591526+0000", 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613358+0000", 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021832949000000001 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613427+0000", 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8955999999999995e-05 2024-09-04T21:55:08.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613495+0000", 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7694999999999995e-05 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32673 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769778926999997, 2024-09-04T21:55:08.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024527538000000002, 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32673 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:08.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589140+0000", 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589143+0000", 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.331e-06 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589153+0000", 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7270000000000002e-06 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589161+0000", 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8149999999999995e-06 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591553+0000", 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023916469999999998 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591558+0000", 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1649999999999997e-06 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613520+0000", 2024-09-04T21:55:08.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021961729999999999 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613591+0000", 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0735000000000006e-05 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613666+0000", 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5754999999999996e-05 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32674 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5128192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769745434999997, 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024653952999999999, 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32674 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589173+0000", 2024-09-04T21:55:08.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000005e-07 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589186+0000", 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2972999999999999e-05 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589204+0000", 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7666999999999999e-05 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589212+0000", 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3999999999999992e-06 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591589+0000", 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023771370000000001 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591594+0000", 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3149999999999999e-06 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613690+0000", 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022095789000000001 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613759+0000", 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9142999999999999e-05 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613826+0000", 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6958999999999994e-05 2024-09-04T21:55:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32675 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5144576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769693015999998, 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024775136, 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32675 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589226+0000", 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0500000000000003e-07 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589229+0000", 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999998e-06 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589239+0000", 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589247+0000", 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0430000000000008e-06 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591622+0000", 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023745099999999998 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591627+0000", 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613851+0000", 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022223613 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613923+0000", 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2203e-05 2024-09-04T21:55:08.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614000+0000", 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32676 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5160960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769659779000001, 2024-09-04T21:55:08.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024914054000000001, 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32676 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589259+0000", 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589276+0000", 2024-09-04T21:55:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7686e-05 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589288+0000", 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1234e-05 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589296+0000", 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5769999999999994e-06 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591654+0000", 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023581769999999999 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591660+0000", 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3109999999999996e-06 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614025+0000", 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022365051 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614099+0000", 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3881000000000006e-05 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614172+0000", 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3498999999999995e-05 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.402 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32677 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5177344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769601747999999, 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025050084, 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32677 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589317+0000", 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5300000000000003e-07 2024-09-04T21:55:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589329+0000", 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1951e-05 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589346+0000", 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7108e-05 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589354+0000", 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2630000000000003e-06 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591674+0000", 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023197980000000001 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591677+0000", 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5909999999999998e-06 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614205+0000", 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022528256999999999 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614287+0000", 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1886999999999997e-05 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614366+0000", 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9475999999999996e-05 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32678 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5193728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769549556000001, 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025172158, 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.408 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32678 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589369+0000", 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8400000000000004e-07 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589373+0000", 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589391+0000", 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8077e-05 2024-09-04T21:55:08.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589400+0000", 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0790000000000007e-06 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591689+0000", 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002288815 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591692+0000", 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.666e-06 2024-09-04T21:55:08.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614392+0000", 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022700287 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614462+0000", 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0468000000000004e-05 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614540+0000", 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8104999999999998e-05 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32679 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5210112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769505623000001, 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025302998, 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.413 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32679 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589413+0000", 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0100000000000004e-07 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589417+0000", 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7689999999999998e-06 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589435+0000", 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8185e-05 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589444+0000", 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4029999999999996e-06 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591704+0000", 2024-09-04T21:55:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022601729999999999 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591708+0000", 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.067e-06 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614571+0000", 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022863270000000002 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614640+0000", 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909e-05 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614715+0000", 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5520999999999997e-05 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32680 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5226496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.769460013, 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025450210000000001, 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32680 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589459+0000", 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-04T21:55:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589462+0000", 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7780000000000001e-06 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589479+0000", 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6651e-05 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589488+0000", 2024-09-04T21:55:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6309999999999992e-06 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591737+0000", 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002249301 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591742+0000", 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4790000000000004e-06 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614747+0000", 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.023004539000000001 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614825+0000", 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8566000000000006e-05 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614908+0000", 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2558999999999993e-05 2024-09-04T21:55:08.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32929 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5120000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759844149000003, 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024146612000000001, 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32929 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599074+0000", 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599077+0000", 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.373e-06 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599083+0000", 2024-09-04T21:55:08.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5060000000000001e-06 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599088+0000", 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0690000000000001e-06 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601461+0000", 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023728299999999998 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601465+0000", 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5129999999999998e-06 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623133+0000", 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021667782 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623178+0000", 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4950000000000002e-05 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623220+0000", 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2076999999999997e-05 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32930 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5136384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759822425000003, 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024238576000000001, 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.429 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32930 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599096+0000", 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:55:08.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599098+0000", 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2979999999999999e-06 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599105+0000", 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5479999999999998e-06 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599110+0000", 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1109999999999999e-06 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601491+0000", 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023813800000000002 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601496+0000", 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:55:08.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623247+0000", 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021751177 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623291+0000", 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4184999999999998e-05 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623334+0000", 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2970999999999999e-05 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32931 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5152768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759800159000001, 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024330243000000001, 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.435 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32931 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.61e-07 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599121+0000", 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.683e-06 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599129+0000", 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6359999999999994e-06 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599134+0000", 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5049999999999998e-06 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601519+0000", 2024-09-04T21:55:08.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002384546 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601521+0000", 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2919999999999998e-06 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623360+0000", 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021839164000000001 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623405+0000", 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4243999999999999e-05 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623448+0000", 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3411999999999997e-05 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32932 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5169152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.440 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759775124999997, 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024418507999999998, 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32932 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599144+0000", 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4799999999999997e-07 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599153+0000", 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9930000000000001e-06 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599161+0000", 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0900000000000005e-06 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599167+0000", 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601543+0000", 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023758540000000002 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601547+0000", 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1019999999999999e-06 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623474+0000", 2024-09-04T21:55:08.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021927782 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623519+0000", 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4263e-05 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623561+0000", 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2483e-05 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32933 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5185536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759741030000001, 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024496766999999999, 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32933 2024-09-04T21:55:08.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-07 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599180+0000", 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599188+0000", 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599203+0000", 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5099999999999999e-05 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601573+0000", 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002369566 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601577+0000", 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5090000000000004e-06 2024-09-04T21:55:08.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623587+0000", 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022009635999999999 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623631+0000", 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4605999999999999e-05 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623674+0000", 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2274000000000002e-05 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32934 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5201920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759704857999999, 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024572765999999999, 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32934 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599214+0000", 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599217+0000", 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.095e-06 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599224+0000", 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6760000000000007e-06 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599230+0000", 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8039999999999998e-06 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601603+0000", 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023727090000000002 2024-09-04T21:55:08.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601608+0000", 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-06 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623700+0000", 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022092219999999999 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623744+0000", 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4013e-05 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623786+0000", 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1749000000000003e-05 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32935 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5218304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:08.456 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759678573999999, 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024658065999999999, 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32935 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599240+0000", 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3500000000000002e-07 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599249+0000", 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599258+0000", 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7069999999999998e-06 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599264+0000", 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4790000000000001e-06 2024-09-04T21:55:08.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601634+0000", 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023694020000000001 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601638+0000", 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623812+0000", 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022173986 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623856+0000", 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3696000000000003e-05 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623897+0000", 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1530999999999999e-05 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.461 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32936 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5234688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.759643222999998, 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024740649, 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.462 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32936 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2099999999999997e-07 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599278+0000", 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8870000000000001e-06 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599285+0000", 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4449999999999999e-06 2024-09-04T21:55:08.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599291+0000", 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601668+0000", 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023769379999999999 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601673+0000", 2024-09-04T21:55:08.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8210000000000001e-06 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623927+0000", 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022253536000000001 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623972+0000", 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5216999999999998e-05 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.624015+0000", 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3519000000000001e-05 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.467 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:08.468 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:08.469 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:08.465+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:08.488 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.5 to 1 (10000) 2024-09-04T21:55:08.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-04T21:55:08.569 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:08.565+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:08.613 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:08.613 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:08.613 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:08.613 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:08.625 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone25 from 0x984aaa8 (overlap 0x161a40d) to 0x0 2024-09-04T21:55:08.628 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:08.671 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:08.669+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:08.750 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:08.751 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:08.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:08.795 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-09-04T21:55:08.795 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:08.793+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:08.872 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:08.869+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:08.905 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:08.905 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:08.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.401777961000001, 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.906 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:08.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.910 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.057001575000001, 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:08.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.05697297, 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.915 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:08.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:08.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:08.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056952933000002, 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.920 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:08.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:08.925 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056933710999999, 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:08.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:08.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056914920000001, 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:08.930 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056882978000001, 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:08.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:08.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056854448999999, 2024-09-04T21:55:08.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:08.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056834838, 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:08.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056816545, 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:08.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056793730999999, 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:08.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056774594, 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.961 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:08.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056753927999999, 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056723151, 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056703783, 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:08.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:08.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056679824, 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:08.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056632574999998, 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.986 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:08.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056606897000002, 2024-09-04T21:55:08.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:08.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056572871, 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:08.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:08.999 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.056548469999999, 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:09.000 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:09.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:09.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:09.004 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:09.006 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:09.006 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:09.001+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:09.106 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:09.105+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112266 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.179940+0000", 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0076748320000000004, 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0077004550000000001, 2024-09-04T21:55:09.194 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112266 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179940+0000", 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179940+0000", 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179941+0000", 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3399999999999997e-07 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179945+0000", 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4379999999999997e-06 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179954+0000", 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2010000000000005e-06 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179960+0000", 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8080000000000001e-06 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.197 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111037 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182124+0000", 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0054901109999999998, 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055567259999999997, 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.198 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111037 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182124+0000", 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182124+0000", 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182125+0000", 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3799999999999998e-07 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182126+0000", 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.409e-06 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182130+0000", 2024-09-04T21:55:09.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7179999999999998e-06 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182133+0000", 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8399999999999999e-06 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111285 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183061+0000", 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0045534490000000002, 2024-09-04T21:55:09.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0046482709999999998, 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111285 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183061+0000", 2024-09-04T21:55:09.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183061+0000", 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183062+0000", 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5799999999999999e-07 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183064+0000", 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5600000000000001e-06 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183073+0000", 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4789999999999996e-06 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183078+0000", 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2100000000000001e-06 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112268 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.179969+0000", 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0076456359999999999, 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0077666239999999998, 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.205 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112268 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179969+0000", 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179969+0000", 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179969+0000", 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5400000000000001e-07 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179972+0000", 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.43e-06 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179980+0000", 2024-09-04T21:55:09.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0050000000000004e-06 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.179985+0000", 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5520000000000004e-06 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111039 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182138+0000", 2024-09-04T21:55:09.208 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0054764430000000001, 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056448879999999998, 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111039 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182138+0000", 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182138+0000", 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182139+0000", 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1370000000000001e-06 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182141+0000", 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4219999999999999e-06 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182144+0000", 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.19e-06 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182147+0000", 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9270000000000001e-06 2024-09-04T21:55:09.211 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111286 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183086+0000", 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0045281169999999999, 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047479940000000002, 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.212 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111286 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183086+0000", 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183086+0000", 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183087+0000", 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8599999999999998e-07 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183089+0000", 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5639999999999999e-06 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183097+0000", 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8509999999999999e-06 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183102+0000", 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9450000000000001e-06 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112271 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.180032+0000", 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0075823619999999996, 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007851986, 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112271 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.216 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180032+0000", 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180032+0000", 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180033+0000", 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8800000000000002e-07 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180035+0000", 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3939999999999999e-06 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180042+0000", 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4470000000000003e-06 2024-09-04T21:55:09.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180046+0000", 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9450000000000003e-06 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111041 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182159+0000", 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0054559439999999999, 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057702539999999998, 2024-09-04T21:55:09.219 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111041 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182159+0000", 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182159+0000", 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182159+0000", 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1100000000000002e-07 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182160+0000", 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.531e-06 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182164+0000", 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7280000000000001e-06 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182167+0000", 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6510000000000001e-06 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111288 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183110+0000", 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0045046260000000003, 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048448659999999998, 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111288 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183110+0000", 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183110+0000", 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183110+0000", 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183113+0000", 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183119+0000", 2024-09-04T21:55:09.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3949999999999998e-06 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183126+0000", 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7769999999999997e-06 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112273 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.180053+0000", 2024-09-04T21:55:09.226 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0075619199999999998, 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0079322560000000004, 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112273 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180053+0000", 2024-09-04T21:55:09.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180053+0000", 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180053+0000", 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.15e-07 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180054+0000", 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4130000000000001e-06 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180058+0000", 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5949999999999999e-06 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180074+0000", 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6359e-05 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.229 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111043 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182171+0000", 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.005443926, 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0058602100000000002, 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.230 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111043 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182171+0000", 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182171+0000", 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182171+0000", 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182172+0000", 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1650000000000001e-06 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182175+0000", 2024-09-04T21:55:09.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182180+0000", 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5839999999999996e-06 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111290 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183134+0000", 2024-09-04T21:55:09.233 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0044810199999999996, 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0049389359999999997, 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111290 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.234 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183134+0000", 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183134+0000", 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183134+0000", 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.06e-07 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183136+0000", 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3939999999999999e-06 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183145+0000", 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3399999999999998e-06 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183150+0000", 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112275 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.180083+0000", 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0075314559999999997, 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0080353379999999995, 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.240 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112275 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180083+0000", 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180083+0000", 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180084+0000", 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4400000000000001e-07 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180086+0000", 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6220000000000001e-06 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180093+0000", 2024-09-04T21:55:09.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2339999999999997e-06 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180099+0000", 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9410000000000004e-06 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111046 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182191+0000", 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0054231310000000003, 2024-09-04T21:55:09.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0059727449999999998, 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111046 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182191+0000", 2024-09-04T21:55:09.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182191+0000", 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182192+0000", 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5800000000000001e-07 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182193+0000", 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4249999999999999e-06 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182197+0000", 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.461e-06 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182199+0000", 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.717e-06 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.246 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111291 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183155+0000", 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0044598720000000001, 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0050512270000000001, 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.247 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111291 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183155+0000", 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183155+0000", 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183155+0000", 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.91e-07 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183156+0000", 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5149999999999999e-06 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183162+0000", 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1259999999999997e-06 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183164+0000", 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7329999999999999e-06 2024-09-04T21:55:09.250 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112277 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.180107+0000", 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0075072539999999997, 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.008126075, 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.251 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112277 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180107+0000", 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180107+0000", 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180108+0000", 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8500000000000002e-07 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180110+0000", 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3360000000000002e-06 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180129+0000", 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9055999999999998e-05 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180141+0000", 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1647e-05 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111048 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182203+0000", 2024-09-04T21:55:09.254 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0054115860000000003, 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060567260000000001, 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111048 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182203+0000", 2024-09-04T21:55:09.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182203+0000", 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182203+0000", 2024-09-04T21:55:09.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4699999999999998e-07 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182214+0000", 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1236999999999999e-05 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182219+0000", 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0879999999999997e-06 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182222+0000", 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8279999999999996e-06 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111292 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183168+0000", 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0044462599999999996, 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0051325110000000002, 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.258 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111292 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183168+0000", 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183168+0000", 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183168+0000", 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-07 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183170+0000", 2024-09-04T21:55:09.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.246e-06 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183173+0000", 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-06 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183176+0000", 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8080000000000001e-06 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.261 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:112279 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.180216+0000", 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0073988020000000003, 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0081318810000000005, 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 112279 2024-09-04T21:55:09.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180216+0000", 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180216+0000", 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180217+0000", 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7499999999999999e-07 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180220+0000", 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3189999999999999e-06 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180228+0000", 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5159999999999994e-06 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.180235+0000", 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.387e-06 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111050 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182230+0000", 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0053848639999999996, 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061576679999999998, 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.265 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111050 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182230+0000", 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182230+0000", 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182230+0000", 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182232+0000", 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0640000000000001e-06 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182240+0000", 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6240000000000002e-06 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182245+0000", 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9609999999999996e-06 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111293 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183179+0000", 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0044352539999999996, 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052427619999999998, 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111293 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183179+0000", 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183179+0000", 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183179+0000", 2024-09-04T21:55:09.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1400000000000001e-07 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183181+0000", 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.147e-06 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183184+0000", 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1520000000000001e-06 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183186+0000", 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.497e-06 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111053 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182266+0000", 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0053483419999999999, 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0062012530000000003, 2024-09-04T21:55:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111053 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182266+0000", 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182266+0000", 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182267+0000", 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182270+0000", 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6819999999999999e-06 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182276+0000", 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6680000000000004e-06 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182282+0000", 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4369999999999998e-06 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.275 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111294 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183190+0000", 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.00442488, 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0053099669999999996, 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.276 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111294 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183190+0000", 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183190+0000", 2024-09-04T21:55:09.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183190+0000", 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183191+0000", 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.159e-06 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183194+0000", 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1769999999999998e-06 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183197+0000", 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.633e-06 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:110992 3.1d 3:b8c56fe1:::rbd_data.1188b9e2364f.0000000000000002:1a [read 1353728~1024] snapc 0=[] RETRY=7 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181470+0000", 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0061446770000000003, 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070768970000000004, 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 110992 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181470+0000", 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181470+0000", 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181471+0000", 2024-09-04T21:55:09.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3500000000000005e-07 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181477+0000", 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4980000000000003e-06 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181487+0000", 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181493+0000", 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3510000000000001e-06 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188458+0000", 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0069653789999999998 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188546+0000", 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8205e-05 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.283 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111055 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182298+0000", 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.005317011, 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0062864380000000001, 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.284 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111055 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182298+0000", 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182298+0000", 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182298+0000", 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0299999999999999e-07 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182301+0000", 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182308+0000", 2024-09-04T21:55:09.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9630000000000001e-06 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182314+0000", 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5740000000000003e-06 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111295 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183200+0000", 2024-09-04T21:55:09.287 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.004414439, 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0054200139999999999, 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111295 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183200+0000", 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183200+0000", 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183200+0000", 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1199999999999999e-07 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183201+0000", 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0720000000000001e-06 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183205+0000", 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.134e-06 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183207+0000", 2024-09-04T21:55:09.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5569999999999998e-06 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:110994 3.1d 3:b8c56fe1:::rbd_data.1188b9e2364f.0000000000000002:1a [read 1358848~1024] snapc 0=[] RETRY=7 ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181503+0000", 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0061116620000000003, 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0071678150000000001, 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:09.291 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 110994 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181503+0000", 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181503+0000", 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181503+0000", 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0100000000000005e-07 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181505+0000", 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7999999999999999e-06 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181509+0000", 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181513+0000", 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2289999999999999e-06 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188640+0000", 2024-09-04T21:55:09.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0071278169999999998 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111057 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182338+0000", 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0052769899999999996, 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0063631399999999998, 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111057 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182338+0000", 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182338+0000", 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182338+0000", 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0900000000000001e-07 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182341+0000", 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6000000000000001e-06 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182348+0000", 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9759999999999998e-06 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182353+0000", 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1250000000000001e-06 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111296 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183210+0000", 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0044041990000000001, 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055288660000000003, 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111296 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.299 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183210+0000", 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183210+0000", 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183211+0000", 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1500000000000001e-07 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183225+0000", 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4242e-05 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183234+0000", 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183238+0000", 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4660000000000001e-06 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:110996 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181532+0000", 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0060823350000000003, 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007237205, 2024-09-04T21:55:09.302 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 110996 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181532+0000", 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181532+0000", 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181533+0000", 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181537+0000", 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181545+0000", 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7830000000000004e-06 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181551+0000", 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9900000000000002e-06 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.305 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111059 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182369+0000", 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0052455840000000002, 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006436382, 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111059 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182369+0000", 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182369+0000", 2024-09-04T21:55:09.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182370+0000", 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7000000000000004e-07 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182372+0000", 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.419e-06 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182379+0000", 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6529999999999997e-06 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182385+0000", 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9699999999999996e-06 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.309 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111297 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183243+0000", 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0043715910000000002, 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056070399999999998, 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111297 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183243+0000", 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183243+0000", 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183278+0000", 2024-09-04T21:55:09.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4993999999999997e-05 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183284+0000", 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0650000000000004e-06 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183297+0000", 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2904e-05 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183305+0000", 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4130000000000006e-06 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:110998 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181561+0000", 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.006053503, 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007319926, 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.313 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 110998 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181561+0000", 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181561+0000", 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181562+0000", 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0500000000000004e-07 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181563+0000", 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6810000000000001e-06 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181567+0000", 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1509999999999997e-06 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181570+0000", 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0580000000000002e-06 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111061 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182402+0000", 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0052125670000000004, 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0065172149999999998, 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111061 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182402+0000", 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182402+0000", 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182403+0000", 2024-09-04T21:55:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8699999999999995e-07 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182405+0000", 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.283e-06 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182412+0000", 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3900000000000004e-06 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182425+0000", 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2295e-05 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111298 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183322+0000", 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0042928150000000002, 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0056743890000000002, 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111298 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183322+0000", 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183322+0000", 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183323+0000", 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3600000000000002e-07 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183326+0000", 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.056e-06 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183334+0000", 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0390000000000005e-06 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183340+0000", 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111000 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181591+0000", 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0060235760000000001, 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074483079999999998, 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111000 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181591+0000", 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181591+0000", 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181591+0000", 2024-09-04T21:55:09.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.82e-07 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181593+0000", 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.773e-06 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181597+0000", 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9299999999999996e-06 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181602+0000", 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4320000000000001e-06 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111063 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182512+0000", 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0051024069999999998, 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0065708609999999999, 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111063 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182512+0000", 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182512+0000", 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182513+0000", 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1099999999999995e-07 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182516+0000", 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.366e-06 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182524+0000", 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182531+0000", 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7630000000000003e-06 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111299 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183358+0000", 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0042564480000000003, 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0057658689999999999, 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111299 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183358+0000", 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183358+0000", 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183359+0000", 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8100000000000002e-07 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183362+0000", 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2049999999999998e-06 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183369+0000", 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2150000000000004e-06 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183375+0000", 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2719999999999997e-06 2024-09-04T21:55:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111002 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181625+0000", 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0059894370000000002, 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0075313929999999999, 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111002 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181625+0000", 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181625+0000", 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181626+0000", 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4099999999999999e-07 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181627+0000", 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.708e-06 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181631+0000", 2024-09-04T21:55:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9720000000000003e-06 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181634+0000", 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.897e-06 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111065 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182543+0000", 2024-09-04T21:55:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0050719700000000003, 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0066616210000000004, 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111065 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182543+0000", 2024-09-04T21:55:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182543+0000", 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182543+0000", 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7300000000000005e-07 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182546+0000", 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5220000000000002e-06 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182553+0000", 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6930000000000001e-06 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182558+0000", 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6489999999999996e-06 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111300 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183397+0000", 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0042174609999999996, 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0058585970000000001, 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111300 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183397+0000", 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183397+0000", 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183398+0000", 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9800000000000004e-07 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183400+0000", 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3080000000000002e-06 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183407+0000", 2024-09-04T21:55:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8070000000000002e-06 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183418+0000", 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1097999999999999e-05 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111004 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181651+0000", 2024-09-04T21:55:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.005963838, 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007655635, 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111004 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181651+0000", 2024-09-04T21:55:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181651+0000", 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181651+0000", 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181653+0000", 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.911e-06 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181658+0000", 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2490000000000003e-06 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181661+0000", 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1839999999999999e-06 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111067 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182566+0000", 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0050488870000000002, 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006793655, 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111067 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182566+0000", 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182566+0000", 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182566+0000", 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7e-07 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182568+0000", 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.159e-06 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182574+0000", 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9780000000000002e-06 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182579+0000", 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8319999999999996e-06 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111301 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183426+0000", 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.004188237, 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0059827400000000003, 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111301 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183426+0000", 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183426+0000", 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183427+0000", 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0900000000000001e-07 2024-09-04T21:55:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183430+0000", 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6649999999999999e-06 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183436+0000", 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7070000000000003e-06 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183441+0000", 2024-09-04T21:55:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9840000000000001e-06 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111005 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181666+0000", 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0059487689999999996, 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0077966859999999997, 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111005 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181666+0000", 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181666+0000", 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181666+0000", 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.23e-07 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181667+0000", 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.271e-06 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181670+0000", 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1999999999999999e-06 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181674+0000", 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4999999999999999e-06 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111069 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182604+0000", 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0050100680000000003, 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0069053320000000001, 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111069 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182604+0000", 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182604+0000", 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182605+0000", 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.46e-07 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182607+0000", 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.742e-06 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182611+0000", 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2710000000000003e-06 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182614+0000", 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8909999999999999e-06 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111302 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183462+0000", 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0041527719999999999, 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060771310000000004, 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111302 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183462+0000", 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183462+0000", 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187708+0000", 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042461299999999999 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187713+0000", 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2190000000000003e-06 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187723+0000", 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8239999999999995e-06 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187740+0000", 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6719999999999999e-05 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111006 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181693+0000", 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0059211660000000003, 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0078899190000000004, 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111006 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181693+0000", 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181693+0000", 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181694+0000", 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.51e-07 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181696+0000", 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6750000000000001e-06 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181699+0000", 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8970000000000001e-06 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181702+0000", 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8420000000000001e-06 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111071 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182620+0000", 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0049946280000000001, 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0069983229999999999, 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111071 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182620+0000", 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182620+0000", 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182621+0000", 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182623+0000", 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.137e-06 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182630+0000", 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3359999999999997e-06 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182635+0000", 2024-09-04T21:55:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7990000000000001e-06 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111303 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187764+0000", 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9998507, 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019027110000000001, 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111303 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187764+0000", 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187764+0000", 2024-09-04T21:55:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187764+0000", 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0100000000000005e-07 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187767+0000", 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0089999999999999e-06 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187774+0000", 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0439999999999998e-06 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187779+0000", 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1710000000000001e-06 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111008 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181727+0000", 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.005887682, 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0079818149999999997, 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111008 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181727+0000", 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181727+0000", 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181727+0000", 2024-09-04T21:55:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9699999999999996e-07 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181730+0000", 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2369999999999999e-06 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181745+0000", 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5673999999999999e-05 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181752+0000", 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2559999999999999e-06 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111074 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182642+0000", 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0049729680000000004, 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070967950000000004, 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111074 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182642+0000", 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182642+0000", 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182642+0000", 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182658+0000", 2024-09-04T21:55:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5519000000000002e-05 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182669+0000", 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.151e-05 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182674+0000", 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3260000000000003e-06 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111304 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187787+0000", 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9998269, 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019893440000000001, 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111304 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187787+0000", 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187787+0000", 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187788+0000", 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7199999999999999e-07 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187790+0000", 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0980000000000001e-06 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187796+0000", 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0349999999999998e-06 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187801+0000", 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6979999999999997e-06 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111010 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181771+0000", 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0058431439999999998, 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0080580659999999991, 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111010 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181771+0000", 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181771+0000", 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181772+0000", 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5000000000000003e-07 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181774+0000", 2024-09-04T21:55:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2680000000000001e-06 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181779+0000", 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9350000000000002e-06 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181783+0000", 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9829999999999998e-06 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111076 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182683+0000", 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.004931164, 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0071734759999999998, 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111076 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182683+0000", 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182683+0000", 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182684+0000", 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7800000000000001e-07 2024-09-04T21:55:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182686+0000", 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4760000000000001e-06 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182693+0000", 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5579999999999997e-06 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182698+0000", 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1760000000000001e-06 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111305 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187822+0000", 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9997931, 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020775110000000002, 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111305 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187822+0000", 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187822+0000", 2024-09-04T21:55:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187822+0000", 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7100000000000002e-07 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187824+0000", 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3479999999999998e-06 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187831+0000", 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3210000000000001e-06 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187836+0000", 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9219999999999996e-06 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111011 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181806+0000", 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0058083290000000001, 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0081212739999999995, 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111011 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181806+0000", 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181806+0000", 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181807+0000", 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3799999999999998e-07 2024-09-04T21:55:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181808+0000", 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.44e-06 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181812+0000", 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6899999999999998e-06 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181815+0000", 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.841e-06 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111262 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182707+0000", 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.004907865, 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0072616019999999998, 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111262 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182707+0000", 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182707+0000", 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182707+0000", 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.39e-07 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182709+0000", 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182713+0000", 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9689999999999996e-06 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182716+0000", 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9679999999999998e-06 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111306 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187853+0000", 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9997621, 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002162841, 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111306 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187853+0000", 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187853+0000", 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187853+0000", 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187855+0000", 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2469999999999998e-06 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187861+0000", 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0900000000000001e-06 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187866+0000", 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8119999999999998e-06 2024-09-04T21:55:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111012 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181818+0000", 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0057961130000000003, 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0082325929999999999, 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111012 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181818+0000", 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181818+0000", 2024-09-04T21:55:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181819+0000", 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-07 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181820+0000", 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2610000000000001e-06 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181823+0000", 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2270000000000002e-06 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181826+0000", 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7920000000000001e-06 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111264 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182734+0000", 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0048803639999999999, 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073565940000000002, 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111264 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182734+0000", 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182734+0000", 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182735+0000", 2024-09-04T21:55:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7899999999999997e-07 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182737+0000", 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1569999999999998e-06 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182749+0000", 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1481000000000001e-05 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182754+0000", 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8549999999999998e-06 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111307 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187883+0000", 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9997315, 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022371140000000001, 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111307 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187883+0000", 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187883+0000", 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187883+0000", 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.58e-07 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187886+0000", 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3250000000000002e-06 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187892+0000", 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.127e-06 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187898+0000", 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5690000000000001e-06 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111014 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181839+0000", 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0057751390000000003, 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0083201330000000004, 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111014 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181839+0000", 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181839+0000", 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181840+0000", 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181841+0000", 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.381e-06 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181845+0000", 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4970000000000001e-06 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181847+0000", 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7640000000000001e-06 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111266 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182761+0000", 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0048530999999999999, 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074471700000000004, 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111266 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182761+0000", 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182761+0000", 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182762+0000", 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182764+0000", 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2699999999999999e-06 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182770+0000", 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9619999999999998e-06 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182776+0000", 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6949999999999997e-06 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111308 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187914+0000", 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9997005, 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023235970000000002, 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111308 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187914+0000", 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187914+0000", 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187914+0000", 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6800000000000001e-07 2024-09-04T21:55:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187929+0000", 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.501e-05 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187940+0000", 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0125999999999999e-05 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187945+0000", 2024-09-04T21:55:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0139999999999998e-06 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111016 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181851+0000", 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0057632810000000003, 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0084268510000000008, 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111016 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181851+0000", 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181851+0000", 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181852+0000", 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181853+0000", 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2020000000000001e-06 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181856+0000", 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0620000000000001e-06 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181859+0000", 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8080000000000001e-06 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111268 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182782+0000", 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0048321229999999998, 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0075349459999999998, 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111268 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182782+0000", 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182782+0000", 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182783+0000", 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1300000000000001e-07 2024-09-04T21:55:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182785+0000", 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2189999999999998e-06 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182789+0000", 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8039999999999999e-06 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182792+0000", 2024-09-04T21:55:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8540000000000001e-06 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111309 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.187973+0000", 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9996414, 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023838459999999998, 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111309 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187973+0000", 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187973+0000", 2024-09-04T21:55:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187973+0000", 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4000000000000002e-07 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187976+0000", 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5569999999999998e-06 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187983+0000", 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6969999999999996e-06 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.187988+0000", 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0300000000000001e-06 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111018 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181872+0000", 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0057425339999999997, 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0086319510000000006, 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111018 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181872+0000", 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181872+0000", 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181872+0000", 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5499999999999999e-07 2024-09-04T21:55:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181874+0000", 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.53e-06 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181877+0000", 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4139999999999998e-06 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181880+0000", 2024-09-04T21:55:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8200000000000001e-06 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111269 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182796+0000", 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0048186720000000004, 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0077507310000000003, 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111269 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182796+0000", 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182796+0000", 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182796+0000", 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2600000000000001e-07 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182797+0000", 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182801+0000", 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2210000000000002e-06 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182804+0000", 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9790000000000002e-06 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111310 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188014+0000", 2024-09-04T21:55:09.445 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9996004, 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025814119999999999, 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111310 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188014+0000", 2024-09-04T21:55:09.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188014+0000", 2024-09-04T21:55:09.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188015+0000", 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6300000000000005e-07 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188017+0000", 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.605e-06 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188024+0000", 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7070000000000003e-06 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188029+0000", 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1370000000000001e-06 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.449 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111020 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181886+0000", 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0057287370000000002, 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0087400029999999997, 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.450 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111020 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181886+0000", 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181886+0000", 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181886+0000", 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5499999999999998e-07 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181888+0000", 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4339999999999999e-06 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181891+0000", 2024-09-04T21:55:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2380000000000002e-06 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181894+0000", 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7860000000000001e-06 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111270 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182807+0000", 2024-09-04T21:55:09.453 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0048075490000000004, 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007860199, 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111270 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182807+0000", 2024-09-04T21:55:09.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182807+0000", 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182807+0000", 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.22e-07 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182808+0000", 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.238e-06 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182813+0000", 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5619999999999997e-06 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182818+0000", 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8189999999999999e-06 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.456 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111311 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188037+0000", 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9995775, 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026571899999999998, 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.457 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111311 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188037+0000", 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188037+0000", 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188037+0000", 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4400000000000001e-07 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188040+0000", 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.154e-06 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188046+0000", 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9379999999999997e-06 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188051+0000", 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7169999999999996e-06 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111021 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181906+0000", 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0057088540000000002, 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0088279020000000003, 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.460 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111021 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181906+0000", 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181906+0000", 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181906+0000", 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181915+0000", 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3200000000000006e-06 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181920+0000", 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5140000000000003e-06 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181924+0000", 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9199999999999997e-06 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111272 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.463 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182822+0000", 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0047921500000000002, 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0079581459999999993, 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111272 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182822+0000", 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182822+0000", 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182823+0000", 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1699999999999998e-07 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182825+0000", 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0509999999999999e-06 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182831+0000", 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7769999999999999e-06 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182836+0000", 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-04T21:55:09.466 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111312 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188066+0000", 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9995484, 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027473469999999998, 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.467 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111312 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188066+0000", 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188066+0000", 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188067+0000", 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9399999999999995e-07 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188069+0000", 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3089999999999998e-06 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188075+0000", 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2060000000000004e-06 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188082+0000", 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.844e-06 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111023 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181941+0000", 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0056739920000000001, 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089045849999999996, 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.470 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111023 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181941+0000", 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181941+0000", 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181941+0000", 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181943+0000", 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.305e-06 2024-09-04T21:55:09.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181950+0000", 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7139999999999996e-06 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181954+0000", 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4259999999999999e-06 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111273 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182842+0000", 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0047725210000000001, 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0080499279999999996, 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111273 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182842+0000", 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182842+0000", 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182842+0000", 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.75e-07 2024-09-04T21:55:09.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182844+0000", 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9410000000000001e-06 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182850+0000", 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182855+0000", 2024-09-04T21:55:09.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7729999999999999e-06 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111313 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188097+0000", 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9995179, 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028216259999999998, 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111313 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188097+0000", 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188097+0000", 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188097+0000", 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7800000000000001e-07 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188099+0000", 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1749999999999999e-06 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188106+0000", 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4049999999999997e-06 2024-09-04T21:55:09.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188111+0000", 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2839999999999997e-06 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111025 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181959+0000", 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0056559130000000003, 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089946450000000008, 2024-09-04T21:55:09.480 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111025 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181959+0000", 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181959+0000", 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181959+0000", 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2300000000000001e-07 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181961+0000", 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.393e-06 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181964+0000", 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3409999999999999e-06 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181967+0000", 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8609999999999998e-06 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111276 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182861+0000", 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0047535659999999999, 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0081277109999999993, 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111276 2024-09-04T21:55:09.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182861+0000", 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182861+0000", 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182861+0000", 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7300000000000002e-07 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182863+0000", 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0310000000000001e-06 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182874+0000", 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0282000000000001e-05 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182879+0000", 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2259999999999996e-06 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111314 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188128+0000", 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9994864, 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028863640000000002, 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.487 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111314 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188128+0000", 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188128+0000", 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188129+0000", 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188131+0000", 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3530000000000002e-06 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188139+0000", 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6739999999999997e-06 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188144+0000", 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9089999999999997e-06 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111026 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.181983+0000", 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0056310709999999996, 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0090632999999999998, 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111026 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181983+0000", 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181983+0000", 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181984+0000", 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:09.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181987+0000", 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2890000000000002e-06 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.181996+0000", 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7830000000000004e-06 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182003+0000", 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8589999999999999e-06 2024-09-04T21:55:09.493 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111277 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182885+0000", 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0047290939999999997, 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0081965949999999992, 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.494 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111277 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182885+0000", 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182885+0000", 2024-09-04T21:55:09.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182886+0000", 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1800000000000001e-07 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182888+0000", 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1730000000000002e-06 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182894+0000", 2024-09-04T21:55:09.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182899+0000", 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0640000000000001e-06 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111315 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188160+0000", 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.999454, 2024-09-04T21:55:09.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029490850000000002, 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111315 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188160+0000", 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188160+0000", 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188161+0000", 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.58e-07 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188163+0000", 2024-09-04T21:55:09.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2620000000000001e-06 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188169+0000", 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1909999999999997e-06 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188174+0000", 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8160000000000001e-06 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111029 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182026+0000", 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0055882129999999999, 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0091135150000000009, 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111029 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.501 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182026+0000", 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182026+0000", 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182027+0000", 2024-09-04T21:55:09.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5899999999999995e-07 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182030+0000", 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.123e-06 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182038+0000", 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9529999999999999e-06 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182044+0000", 2024-09-04T21:55:09.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0170000000000002e-06 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111280 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182920+0000", 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0046940840000000003, 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0082565880000000005, 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.504 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111280 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182920+0000", 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182920+0000", 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182921+0000", 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1299999999999999e-07 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182935+0000", 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4039000000000001e-05 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182945+0000", 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-04T21:55:09.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182952+0000", 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7039999999999997e-06 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111316 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188190+0000", 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9994245, 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003012317, 2024-09-04T21:55:09.507 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111316 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188190+0000", 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188190+0000", 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188191+0000", 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2199999999999999e-07 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188193+0000", 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-04T21:55:09.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188209+0000", 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6251000000000001e-05 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188214+0000", 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2240000000000003e-06 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111033 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182063+0000", 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0055518299999999998, 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0091724170000000004, 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111033 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182063+0000", 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182063+0000", 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182063+0000", 2024-09-04T21:55:09.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4099999999999998e-07 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182067+0000", 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182072+0000", 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8950000000000003e-06 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182076+0000", 2024-09-04T21:55:09.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6789999999999998e-06 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111282 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182984+0000", 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0046307680000000004, 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0082880839999999994, 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111282 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182984+0000", 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182984+0000", 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182984+0000", 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0200000000000001e-07 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182987+0000", 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6730000000000001e-06 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182995+0000", 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7989999999999994e-06 2024-09-04T21:55:09.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183005+0000", 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8660000000000001e-06 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111317 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188222+0000", 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.999392, 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030786540000000001, 2024-09-04T21:55:09.517 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111317 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188222+0000", 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188222+0000", 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188223+0000", 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0800000000000005e-07 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188225+0000", 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4159999999999998e-06 2024-09-04T21:55:09.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188231+0000", 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8989999999999998e-06 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188236+0000", 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6639999999999997e-06 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111035 3.1d 3.87f6a31d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.182093+0000", 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0055216479999999997, 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0092383750000000001, 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111035 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182093+0000", 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182093+0000", 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182093+0000", 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182095+0000", 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8479999999999999e-06 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182099+0000", 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0629999999999999e-06 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.182102+0000", 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8959999999999999e-06 2024-09-04T21:55:09.523 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111283 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.183026+0000", 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0045884259999999996, 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0083517139999999997, 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.524 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111283 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183026+0000", 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183026+0000", 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183027+0000", 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9100000000000003e-07 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183031+0000", 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2699999999999998e-06 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183042+0000", 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0484e-05 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.183048+0000", 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4710000000000004e-06 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:111318 3.1d 3.25ec79dd (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:09.527 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:09.188249+0000", 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4294967295.9993653, 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003154998, 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 111318 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:09.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188249+0000", 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188249+0000", 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188250+0000", 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188252+0000", 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188258+0000", 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0569999999999998e-06 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:09.188263+0000", 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9420000000000003e-06 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:09.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:09.531 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-04T21:55:09.531 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 95 2024-09-04T21:55:09.531 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:09.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:09.532 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:55:09.533 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.6 to 1 (10000) 2024-09-04T21:55:09.534 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:09.534 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-04T21:55:09.535 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:09.529+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:09.633 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:09.633+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:09.708 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:09.708 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:09.708 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:09.708 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:09.708 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:09.728 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:09.756 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:09.756 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:09.774 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:09.773+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:09.788 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:09.788 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:09.809 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:09.810 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:09.826 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:09.826 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:09.834 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:09.833+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:09.843 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:09.844 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:09.852 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:09.852 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:09.861 INFO:teuthology.orchestra.run.smithi112.stdout:4650 write 0x75af4cd thru 0x75bddcd (0xe901 bytes) 2024-09-04T21:55:09.865 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:09.871 INFO:teuthology.orchestra.run.smithi112.stdout:4655 read 0x1cfb3a9 thru 0x1cfde17 (0x2a6f bytes) 2024-09-04T21:55:09.872 INFO:teuthology.orchestra.run.smithi112.stdout:4656 read 0x74900eb thru 0x7494353 (0x4269 bytes) 2024-09-04T21:55:09.876 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:09.877 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:09.889 INFO:teuthology.orchestra.run.smithi112.stdout:4657 writesame 0xedc156c thru 0xedc85ba data_size 0x1(0x704f bytes) 2024-09-04T21:55:09.889 INFO:teuthology.orchestra.run.smithi112.stdout:4658 write 0xd604a95 thru 0xd60992c (0x4e98 bytes) 2024-09-04T21:55:09.894 INFO:teuthology.orchestra.run.smithi112.stdout:4659 punch from 0xa634273 to 0xa638a4d, (0x47da bytes) 2024-09-04T21:55:09.894 INFO:teuthology.orchestra.run.smithi112.stdout:4660 write 0xb0753d7 thru 0xb07814e (0x2d78 bytes) 2024-09-04T21:55:09.894 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:09.895 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:09.897 INFO:teuthology.orchestra.run.smithi112.stdout:4661 writesame 0x14a74eb thru 0x14a7c56 data_size 0x19(0x76c bytes) 2024-09-04T21:55:09.904 INFO:teuthology.orchestra.run.smithi112.stdout:4662 writesame 0x6596d8b thru 0x65a2401 data_size 0x1(0xb677 bytes) 2024-09-04T21:55:09.916 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:09.916 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:09.930 INFO:teuthology.orchestra.run.smithi112.stdout:4664 write 0xa94ab55 thru 0xa950055 (0x5501 bytes) 2024-09-04T21:55:09.945 INFO:teuthology.orchestra.run.smithi112.stdout:4665 trunc from 0xedc85bb to 0x79b762e 2024-09-04T21:55:09.945 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:09.945+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:09.976 INFO:teuthology.orchestra.run.smithi112.stdout:4667 writesame 0x1c64199 thru 0x1c6a8e7 data_size 0x1(0x674f bytes) 2024-09-04T21:55:10.010 INFO:teuthology.orchestra.run.smithi112.stdout:4668 write 0xdc81c9f thru 0xdc8624e (0x45b0 bytes) 2024-09-04T21:55:10.023 INFO:teuthology.orchestra.run.smithi112.stdout:4669 write 0xe7045a7 thru 0xe70b8fa (0x7354 bytes) 2024-09-04T21:55:10.027 INFO:teuthology.orchestra.run.smithi112.stdout:4671 punch from 0x51cfae1 to 0x51d08a4, (0xdc3 bytes) 2024-09-04T21:55:10.034 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.034 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.036 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:10.033+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:10.037 INFO:teuthology.orchestra.run.smithi112.stdout:4672 trunc from 0xe70b8fb to 0x836eb30 2024-09-04T21:55:10.048 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:10.049 INFO:teuthology.orchestra.run.smithi112.stdout:4673 trunc from 0x836eb30 to 0x85b0135 2024-09-04T21:55:10.056 INFO:teuthology.orchestra.run.smithi112.stdout:4675 read 0x67c11b thru 0x686cc8 (0xabae bytes) 2024-09-04T21:55:10.057 INFO:teuthology.orchestra.run.smithi112.stdout:4676 write 0x7696fd6 thru 0x769f0bd (0x80e8 bytes) 2024-09-04T21:55:10.062 INFO:teuthology.orchestra.run.smithi112.stdout:4677 punch from 0x9c8b6a to 0x9d339d, (0xa833 bytes) 2024-09-04T21:55:10.076 INFO:teuthology.orchestra.run.smithi112.stdout:4678 write 0xb5c4e99 thru 0xb5d444c (0xf5b4 bytes) 2024-09-04T21:55:10.081 INFO:teuthology.orchestra.run.smithi112.stdout:4679 clone 27 order 19 su 262144 sc 2 2024-09-04T21:55:10.137 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:10.137+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:10.199 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.200 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.212 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:10.259 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:10.257+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:10.321 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.321 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.334 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:10.375 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:10.373+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:10.439 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:10.437+0000 7fb55de13640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26493) UID: 0 2024-09-04T21:55:10.490 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.490 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.494 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.7 to 1 (10000) 2024-09-04T21:55:10.500 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:55:10.506 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:10.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-04T21:55:10.524 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:55:10.547 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:10.545+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:10.614 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone26 from 0xb5d444d (overlap 0x79b762e) to 0x3cdbb17 2024-09-04T21:55:10.642 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:10.641+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:10.697 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.698 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.717 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:10.742 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:10.741+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:10.827 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.828 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.841 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:10.843 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:10.841+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:10.845 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-09-04T21:55:10.848 INFO:teuthology.orchestra.run.smithi112.stdout:4680 read 0x28b9c3f thru 0x28c57a5 (0xbb67 bytes) 2024-09-04T21:55:10.851 INFO:teuthology.orchestra.run.smithi112.stdout:4681 trunc from 0xb5d444d to 0xc8d00cc 2024-09-04T21:55:10.865 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:10.865 INFO:teuthology.orchestra.run.smithi112.stdout:4684 writesame 0x6a9edec thru 0x6aa8a4c data_size 0x31(0x9c61 bytes) 2024-09-04T21:55:10.871 INFO:teuthology.orchestra.run.smithi112.stdout:4685 write 0x2374968 thru 0x2383de7 (0xf480 bytes) 2024-09-04T21:55:10.878 INFO:teuthology.orchestra.run.smithi112.stdout:4686 write 0x4d95833 thru 0x4da0c43 (0xb411 bytes) 2024-09-04T21:55:10.885 INFO:teuthology.orchestra.run.smithi112.stdout:4687 write 0xa8ede0d thru 0xa8f24e0 (0x46d4 bytes) 2024-09-04T21:55:10.894 INFO:teuthology.orchestra.run.smithi112.stdout:4688 trunc from 0xc8d00cc to 0xe6c425d 2024-09-04T21:55:10.911 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:55:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:55:10.913 INFO:teuthology.orchestra.run.smithi112.stdout:4689 write 0xeb9ae89 thru 0xeb9e52d (0x36a5 bytes) 2024-09-04T21:55:10.916 INFO:teuthology.orchestra.run.smithi112.stdout:4692 punch from 0xa7a1f9f to 0xa7ad102, (0xb163 bytes) 2024-09-04T21:55:10.916 INFO:teuthology.orchestra.run.smithi112.stdout:4693 writesame 0xb77dc8d thru 0xb78530f data_size 0x3(0x7683 bytes) 2024-09-04T21:55:10.925 INFO:teuthology.orchestra.run.smithi112.stdout:4694 punch from 0xac2f21e to 0xac35159, (0x5f3b bytes) 2024-09-04T21:55:10.925 INFO:teuthology.orchestra.run.smithi112.stdout:4695 punch from 0x3ef6bd1 to 0x3ef6f89, (0x3b8 bytes) 2024-09-04T21:55:10.925 INFO:teuthology.orchestra.run.smithi112.stdout:4696 write 0xdb410d6 thru 0xdb458dd (0x4808 bytes) 2024-09-04T21:55:10.932 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-04T21:55:10.496224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":803613395,"num_objects":1020,"num_object_clones":290,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":293,"num_read":141136,"num_read_kb":508418,"num_write":13998,"num_write_kb":237247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":55066178,"num_keys_recovered":164,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10581,"ondisk_log_size":10581,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":276016,"kb_used_data":39328,"kb_used_omap":12,"kb_used_meta":236531,"kb_avail":565955024,"statfs":{"total":579820584960,"available":579537944576,"internally_reserved":0,"allocated":40271872,"data_stored":297855807,"data_compressed":14477013,"data_compressed_allocated":26718208,"data_compressed_original":290705535,"omap_allocated":12366,"internal_metadata":242208690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47428803,"num_objects":83,"num_object_clones":42,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":47,"num_read":24211,"num_read_kb":28549,"num_write":620,"num_write_kb":24491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030567"},"pg_stats":[{"pgid":"3.1f","version":"77'170","reported_seq":814,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485966+0000","last_change":"2024-09-04T21:54:51.351220+0000","last_active":"2024-09-04T21:55:07.485966+0000","last_peered":"2024-09-04T21:55:07.485966+0000","last_clean":"2024-09-04T21:55:07.485966+0000","last_became_active":"2024-09-04T21:54:51.350505+0000","last_became_peered":"2024-09-04T21:54:51.350505+0000","last_unstale":"2024-09-04T21:55:07.485966+0000","last_undegraded":"2024-09-04T21:55:07.485966+0000","last_fullsized":"2024-09-04T21:55:07.485966+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:48:33.984938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003772475,"stat_sum":{"num_bytes":15729141,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":9,"num_read":985,"num_read_kb":2424,"num_write":181,"num_write_kb":2132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":330234,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"81'208","reported_seq":1948,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.701793+0000","last_change":"2024-09-04T21:53:57.370393+0000","last_active":"2024-09-04T21:55:07.701793+0000","last_peered":"2024-09-04T21:55:07.701793+0000","last_clean":"2024-09-04T21:55:07.701793+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:55:07.701793+0000","last_undegraded":"2024-09-04T21:55:07.701793+0000","last_fullsized":"2024-09-04T21:55:07.701793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:07:09.045775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372790000000002,"stat_sum":{"num_bytes":16454396,"num_objects":32,"num_object_clones":8,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":8,"num_read":1758,"num_read_kb":8942,"num_write":202,"num_write_kb":4987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'405","reported_seq":14103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487294+0000","last_change":"2024-09-04T21:54:42.681915+0000","last_active":"2024-09-04T21:55:08.487294+0000","last_peered":"2024-09-04T21:55:08.487294+0000","last_clean":"2024-09-04T21:55:08.487294+0000","last_became_active":"2024-09-04T21:54:42.681042+0000","last_became_peered":"2024-09-04T21:54:42.681042+0000","last_unstale":"2024-09-04T21:55:08.487294+0000","last_undegraded":"2024-09-04T21:55:08.487294+0000","last_fullsized":"2024-09-04T21:55:08.487294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:47:35.449231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00153718,"stat_sum":{"num_bytes":70048262,"num_objects":33,"num_object_clones":11,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":7,"num_read":14914,"num_read_kb":17060,"num_write":606,"num_write_kb":8024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"82'235","reported_seq":1946,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.722400+0000","last_change":"2024-09-04T21:54:42.680392+0000","last_active":"2024-09-04T21:55:08.722400+0000","last_peered":"2024-09-04T21:55:08.722400+0000","last_clean":"2024-09-04T21:55:08.722400+0000","last_became_active":"2024-09-04T21:54:42.679589+0000","last_became_peered":"2024-09-04T21:54:42.679589+0000","last_unstale":"2024-09-04T21:55:08.722400+0000","last_undegraded":"2024-09-04T21:55:08.722400+0000","last_fullsized":"2024-09-04T21:55:08.722400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:39:37.588120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349371,"stat_sum":{"num_bytes":21274371,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":1679,"num_read_kb":20468,"num_write":235,"num_write_kb":4710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"81'171","reported_seq":1339,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487235+0000","last_change":"2024-09-04T21:54:51.355968+0000","last_active":"2024-09-04T21:55:08.487235+0000","last_peered":"2024-09-04T21:55:08.487235+0000","last_clean":"2024-09-04T21:55:08.487235+0000","last_became_active":"2024-09-04T21:54:51.354934+0000","last_became_peered":"2024-09-04T21:54:51.354934+0000","last_unstale":"2024-09-04T21:55:08.487235+0000","last_undegraded":"2024-09-04T21:55:08.487235+0000","last_fullsized":"2024-09-04T21:55:08.487235+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:41:34.673551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295954,"stat_sum":{"num_bytes":5986096,"num_objects":22,"num_object_clones":8,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":6,"num_read":967,"num_read_kb":5764,"num_write":169,"num_write_kb":5902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2174559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"77'630","reported_seq":11557,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.718823+0000","last_change":"2024-09-04T21:54:13.554263+0000","last_active":"2024-09-04T21:55:08.718823+0000","last_peered":"2024-09-04T21:55:08.718823+0000","last_clean":"2024-09-04T21:55:08.718823+0000","last_became_active":"2024-09-04T21:54:13.481244+0000","last_became_peered":"2024-09-04T21:54:13.481244+0000","last_unstale":"2024-09-04T21:55:08.718823+0000","last_undegraded":"2024-09-04T21:55:08.718823+0000","last_fullsized":"2024-09-04T21:55:08.718823+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:56:40.949724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001385184,"stat_sum":{"num_bytes":33356472,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":11,"num_read":13125,"num_read_kb":42576,"num_write":946,"num_write_kb":8663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":930020,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"81'1063","reported_seq":1987,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.624734+0000","last_change":"2024-09-04T21:54:13.510809+0000","last_active":"2024-09-04T21:55:07.624734+0000","last_peered":"2024-09-04T21:55:07.624734+0000","last_clean":"2024-09-04T21:55:07.624734+0000","last_became_active":"2024-09-04T21:54:13.478356+0000","last_became_peered":"2024-09-04T21:54:13.478356+0000","last_unstale":"2024-09-04T21:55:07.624734+0000","last_undegraded":"2024-09-04T21:55:07.624734+0000","last_fullsized":"2024-09-04T21:55:07.624734+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1063,"log_dups_size":0,"ondisk_log_size":1063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:10:19.924484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032359149999999998,"stat_sum":{"num_bytes":14896953,"num_objects":29,"num_object_clones":8,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":10,"num_read":5877,"num_read_kb":15599,"num_write":1782,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4393,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"77'263","reported_seq":3130,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612791+0000","last_change":"2024-09-04T21:54:51.350327+0000","last_active":"2024-09-04T21:55:07.612791+0000","last_peered":"2024-09-04T21:55:07.612791+0000","last_clean":"2024-09-04T21:55:07.612791+0000","last_became_active":"2024-09-04T21:54:51.349573+0000","last_became_peered":"2024-09-04T21:54:51.349573+0000","last_unstale":"2024-09-04T21:55:07.612791+0000","last_undegraded":"2024-09-04T21:55:07.612791+0000","last_fullsized":"2024-09-04T21:55:07.612791+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:49:24.297704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001549499,"stat_sum":{"num_bytes":23194872,"num_objects":25,"num_object_clones":8,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":8,"num_read":4048,"num_read_kb":24063,"num_write":285,"num_write_kb":7248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3632211,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"77'767","reported_seq":2379,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.613041+0000","last_change":"2024-09-04T21:54:51.353124+0000","last_active":"2024-09-04T21:55:07.613041+0000","last_peered":"2024-09-04T21:55:07.613041+0000","last_clean":"2024-09-04T21:55:07.613041+0000","last_became_active":"2024-09-04T21:54:51.352376+0000","last_became_peered":"2024-09-04T21:54:51.352376+0000","last_unstale":"2024-09-04T21:55:07.613041+0000","last_undegraded":"2024-09-04T21:55:07.613041+0000","last_fullsized":"2024-09-04T21:55:07.613041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:41:22.321182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035280929999999999,"stat_sum":{"num_bytes":31613458,"num_objects":37,"num_object_clones":4,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":13,"num_read":4973,"num_read_kb":21787,"num_write":1160,"num_write_kb":6773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3732324,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'123","reported_seq":1314,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499711+0000","last_change":"2024-09-04T21:54:42.680486+0000","last_active":"2024-09-04T21:55:09.499711+0000","last_peered":"2024-09-04T21:55:09.499711+0000","last_clean":"2024-09-04T21:55:09.499711+0000","last_became_active":"2024-09-04T21:54:42.679389+0000","last_became_peered":"2024-09-04T21:54:42.679389+0000","last_unstale":"2024-09-04T21:55:09.499711+0000","last_undegraded":"2024-09-04T21:55:09.499711+0000","last_fullsized":"2024-09-04T21:55:09.499711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:49:40.549041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086423499999999998,"stat_sum":{"num_bytes":13680299,"num_objects":18,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":1016,"num_read_kb":13715,"num_write":130,"num_write_kb":4823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"77'200","reported_seq":1748,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499897+0000","last_change":"2024-09-04T21:54:31.683707+0000","last_active":"2024-09-04T21:55:09.499897+0000","last_peered":"2024-09-04T21:55:09.499897+0000","last_clean":"2024-09-04T21:55:09.499897+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:55:09.499897+0000","last_undegraded":"2024-09-04T21:55:09.499897+0000","last_fullsized":"2024-09-04T21:55:09.499897+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:29:12.336532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078592199999999999,"stat_sum":{"num_bytes":18886378,"num_objects":40,"num_object_clones":11,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":18,"num_read":1900,"num_read_kb":19234,"num_write":199,"num_write_kb":14334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"77'162","reported_seq":1379,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487788+0000","last_change":"2024-09-04T21:54:31.673826+0000","last_active":"2024-09-04T21:55:08.487788+0000","last_peered":"2024-09-04T21:55:08.487788+0000","last_clean":"2024-09-04T21:55:08.487788+0000","last_became_active":"2024-09-04T21:54:31.582403+0000","last_became_peered":"2024-09-04T21:54:31.582403+0000","last_unstale":"2024-09-04T21:55:08.487788+0000","last_undegraded":"2024-09-04T21:55:08.487788+0000","last_fullsized":"2024-09-04T21:55:08.487788+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:36:22.287956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001035425,"stat_sum":{"num_bytes":9043312,"num_objects":30,"num_object_clones":11,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":1141,"num_read_kb":5869,"num_write":188,"num_write_kb":3810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":529142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493142+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:55:08.493142+0000","last_peered":"2024-09-04T21:55:08.493142+0000","last_clean":"2024-09-04T21:55:08.493142+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:55:08.493142+0000","last_undegraded":"2024-09-04T21:55:08.493142+0000","last_fullsized":"2024-09-04T21:55:08.493142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"77'210","reported_seq":962,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.488090+0000","last_change":"2024-09-04T21:54:51.354834+0000","last_active":"2024-09-04T21:55:08.488090+0000","last_peered":"2024-09-04T21:55:08.488090+0000","last_clean":"2024-09-04T21:55:08.488090+0000","last_became_active":"2024-09-04T21:54:51.353742+0000","last_became_peered":"2024-09-04T21:54:51.353742+0000","last_unstale":"2024-09-04T21:55:08.488090+0000","last_undegraded":"2024-09-04T21:55:08.488090+0000","last_fullsized":"2024-09-04T21:55:08.488090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:21:20.454105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001390173,"stat_sum":{"num_bytes":5219982,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":6,"num_read":862,"num_read_kb":2160,"num_write":205,"num_write_kb":2877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":645779,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493295+0000","last_change":"2024-09-04T21:54:13.481230+0000","last_active":"2024-09-04T21:55:08.493295+0000","last_peered":"2024-09-04T21:55:08.493295+0000","last_clean":"2024-09-04T21:55:08.493295+0000","last_became_active":"2024-09-04T21:54:13.479390+0000","last_became_peered":"2024-09-04T21:54:13.479390+0000","last_unstale":"2024-09-04T21:55:08.493295+0000","last_undegraded":"2024-09-04T21:55:08.493295+0000","last_fullsized":"2024-09-04T21:55:08.493295+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:05.478515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"77'173","reported_seq":2608,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.613090+0000","last_change":"2024-09-04T21:54:31.773042+0000","last_active":"2024-09-04T21:55:07.613090+0000","last_peered":"2024-09-04T21:55:07.613090+0000","last_clean":"2024-09-04T21:55:07.613090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:55:07.613090+0000","last_undegraded":"2024-09-04T21:55:07.613090+0000","last_fullsized":"2024-09-04T21:55:07.613090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:32:32.369703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015346380000000001,"stat_sum":{"num_bytes":10882634,"num_objects":37,"num_object_clones":12,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":9,"num_read":2848,"num_read_kb":19568,"num_write":226,"num_write_kb":9463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499950+0000","last_change":"2024-09-04T21:54:48.374647+0000","last_active":"2024-09-04T21:55:09.499950+0000","last_peered":"2024-09-04T21:55:09.499950+0000","last_clean":"2024-09-04T21:55:09.499950+0000","last_became_active":"2024-09-04T21:54:48.374288+0000","last_became_peered":"2024-09-04T21:54:48.374288+0000","last_unstale":"2024-09-04T21:55:09.499950+0000","last_undegraded":"2024-09-04T21:55:09.499950+0000","last_fullsized":"2024-09-04T21:55:09.499950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:14:06.038769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"77'187","reported_seq":1247,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485820+0000","last_change":"2024-09-04T21:54:13.539266+0000","last_active":"2024-09-04T21:55:07.485820+0000","last_peered":"2024-09-04T21:55:07.485820+0000","last_clean":"2024-09-04T21:55:07.485820+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:55:07.485820+0000","last_undegraded":"2024-09-04T21:55:07.485820+0000","last_fullsized":"2024-09-04T21:55:07.485820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:30:25.920556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091729799999999998,"stat_sum":{"num_bytes":7414563,"num_objects":22,"num_object_clones":6,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":6,"num_read":1540,"num_read_kb":5820,"num_write":189,"num_write_kb":2889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":82,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499504+0000","last_change":"2024-09-04T21:54:48.372640+0000","last_active":"2024-09-04T21:55:09.499504+0000","last_peered":"2024-09-04T21:55:09.499504+0000","last_clean":"2024-09-04T21:55:09.499504+0000","last_became_active":"2024-09-04T21:54:48.372106+0000","last_became_peered":"2024-09-04T21:54:48.372106+0000","last_unstale":"2024-09-04T21:55:09.499504+0000","last_undegraded":"2024-09-04T21:55:09.499504+0000","last_fullsized":"2024-09-04T21:55:09.499504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:18:59.210273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"77'375","reported_seq":8783,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612935+0000","last_change":"2024-09-04T21:54:42.680694+0000","last_active":"2024-09-04T21:55:07.612935+0000","last_peered":"2024-09-04T21:55:07.612935+0000","last_clean":"2024-09-04T21:55:07.612935+0000","last_became_active":"2024-09-04T21:54:42.679683+0000","last_became_peered":"2024-09-04T21:54:42.679683+0000","last_unstale":"2024-09-04T21:55:07.612935+0000","last_undegraded":"2024-09-04T21:55:07.612935+0000","last_fullsized":"2024-09-04T21:55:07.612935+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:00:31.225665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012888800000000001,"stat_sum":{"num_bytes":29219111,"num_objects":28,"num_object_clones":9,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":6,"num_read":9766,"num_read_kb":22284,"num_write":454,"num_write_kb":9513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"77'195","reported_seq":2842,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487886+0000","last_change":"2024-09-04T21:54:51.357970+0000","last_active":"2024-09-04T21:55:08.487886+0000","last_peered":"2024-09-04T21:55:08.487886+0000","last_clean":"2024-09-04T21:55:08.487886+0000","last_became_active":"2024-09-04T21:54:51.357182+0000","last_became_peered":"2024-09-04T21:54:51.357182+0000","last_unstale":"2024-09-04T21:55:08.487886+0000","last_undegraded":"2024-09-04T21:55:08.487886+0000","last_fullsized":"2024-09-04T21:55:08.487886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:28:43.729296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050067700000000004,"stat_sum":{"num_bytes":13631355,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":2488,"num_read_kb":14671,"num_write":215,"num_write_kb":11174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":2060586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.492929+0000","last_change":"2024-09-04T21:54:42.675259+0000","last_active":"2024-09-04T21:55:08.492929+0000","last_peered":"2024-09-04T21:55:08.492929+0000","last_clean":"2024-09-04T21:55:08.492929+0000","last_became_active":"2024-09-04T21:54:42.674803+0000","last_became_peered":"2024-09-04T21:54:42.674803+0000","last_unstale":"2024-09-04T21:55:08.492929+0000","last_undegraded":"2024-09-04T21:55:08.492929+0000","last_fullsized":"2024-09-04T21:55:08.492929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:09:21.260817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'250","reported_seq":2109,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485329+0000","last_change":"2024-09-04T21:54:16.342878+0000","last_active":"2024-09-04T21:55:07.485329+0000","last_peered":"2024-09-04T21:55:07.485329+0000","last_clean":"2024-09-04T21:55:07.485329+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:55:07.485329+0000","last_undegraded":"2024-09-04T21:55:07.485329+0000","last_fullsized":"2024-09-04T21:55:07.485329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:51:50.337051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010876340000000001,"stat_sum":{"num_bytes":31643640,"num_objects":43,"num_object_clones":11,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":2751,"num_read_kb":9058,"num_write":490,"num_write_kb":20359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485467+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:55:07.485467+0000","last_peered":"2024-09-04T21:55:07.485467+0000","last_clean":"2024-09-04T21:55:07.485467+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:55:07.485467+0000","last_undegraded":"2024-09-04T21:55:07.485467+0000","last_fullsized":"2024-09-04T21:55:07.485467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"76'344","reported_seq":2836,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.564723+0000","last_change":"2024-09-04T21:54:51.354264+0000","last_active":"2024-09-04T21:55:08.564723+0000","last_peered":"2024-09-04T21:55:08.564723+0000","last_clean":"2024-09-04T21:55:08.564723+0000","last_became_active":"2024-09-04T21:54:51.353319+0000","last_became_peered":"2024-09-04T21:54:51.353319+0000","last_unstale":"2024-09-04T21:55:08.564723+0000","last_undegraded":"2024-09-04T21:55:08.564723+0000","last_fullsized":"2024-09-04T21:55:08.564723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:06.335488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396415,"stat_sum":{"num_bytes":42089206,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":3633,"num_read_kb":25388,"num_write":498,"num_write_kb":5581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3580019,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499453+0000","last_change":"2024-09-04T21:54:48.373076+0000","last_active":"2024-09-04T21:55:09.499453+0000","last_peered":"2024-09-04T21:55:09.499453+0000","last_clean":"2024-09-04T21:55:09.499453+0000","last_became_active":"2024-09-04T21:54:48.372613+0000","last_became_peered":"2024-09-04T21:54:48.372613+0000","last_unstale":"2024-09-04T21:55:09.499453+0000","last_undegraded":"2024-09-04T21:55:09.499453+0000","last_fullsized":"2024-09-04T21:55:09.499453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:48:13.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"77'284","reported_seq":2738,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612834+0000","last_change":"2024-09-04T21:53:57.369713+0000","last_active":"2024-09-04T21:55:07.612834+0000","last_peered":"2024-09-04T21:55:07.612834+0000","last_clean":"2024-09-04T21:55:07.612834+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:55:07.612834+0000","last_undegraded":"2024-09-04T21:55:07.612834+0000","last_fullsized":"2024-09-04T21:55:07.612834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:28.880126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026956770000000001,"stat_sum":{"num_bytes":27973372,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":3388,"num_read_kb":13649,"num_write":277,"num_write_kb":4740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485604+0000","last_change":"2024-09-04T21:54:42.676579+0000","last_active":"2024-09-04T21:55:07.485604+0000","last_peered":"2024-09-04T21:55:07.485604+0000","last_clean":"2024-09-04T21:55:07.485604+0000","last_became_active":"2024-09-04T21:54:42.676301+0000","last_became_peered":"2024-09-04T21:54:42.676301+0000","last_unstale":"2024-09-04T21:55:07.485604+0000","last_undegraded":"2024-09-04T21:55:07.485604+0000","last_fullsized":"2024-09-04T21:55:07.485604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:42:21.618311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":192,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499421+0000","last_change":"2024-09-04T21:54:48.383973+0000","last_active":"2024-09-04T21:55:09.499421+0000","last_peered":"2024-09-04T21:55:09.499421+0000","last_clean":"2024-09-04T21:55:09.499421+0000","last_became_active":"2024-09-04T21:54:48.374571+0000","last_became_peered":"2024-09-04T21:54:48.374571+0000","last_unstale":"2024-09-04T21:55:09.499421+0000","last_undegraded":"2024-09-04T21:55:09.499421+0000","last_fullsized":"2024-09-04T21:55:09.499421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:54:44.340649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"77'427","reported_seq":2724,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612829+0000","last_change":"2024-09-04T21:54:34.607151+0000","last_active":"2024-09-04T21:55:07.612829+0000","last_peered":"2024-09-04T21:55:07.612829+0000","last_clean":"2024-09-04T21:55:07.612829+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:55:07.612829+0000","last_undegraded":"2024-09-04T21:55:07.612829+0000","last_fullsized":"2024-09-04T21:55:07.612829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:46:29.393261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001397155,"stat_sum":{"num_bytes":20789094,"num_objects":32,"num_object_clones":10,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":6,"num_read":3848,"num_read_kb":13427,"num_write":567,"num_write_kb":6154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"77'196","reported_seq":1730,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487255+0000","last_change":"2024-09-04T21:53:57.346105+0000","last_active":"2024-09-04T21:55:08.487255+0000","last_peered":"2024-09-04T21:55:08.487255+0000","last_clean":"2024-09-04T21:55:08.487255+0000","last_became_active":"2024-09-04T21:53:55.345842+0000","last_became_peered":"2024-09-04T21:53:55.345842+0000","last_unstale":"2024-09-04T21:55:08.487255+0000","last_undegraded":"2024-09-04T21:55:08.487255+0000","last_fullsized":"2024-09-04T21:55:08.487255+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:40:38.827361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094585800000000005,"stat_sum":{"num_bytes":13837482,"num_objects":35,"num_object_clones":13,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":13,"num_read":1422,"num_read_kb":12785,"num_write":185,"num_write_kb":5335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"82'460","reported_seq":2175,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493669+0000","last_change":"2024-09-04T21:53:57.339901+0000","last_active":"2024-09-04T21:55:08.493669+0000","last_peered":"2024-09-04T21:55:08.493669+0000","last_clean":"2024-09-04T21:55:08.493669+0000","last_became_active":"2024-09-04T21:53:55.355393+0000","last_became_peered":"2024-09-04T21:53:55.355393+0000","last_unstale":"2024-09-04T21:55:08.493669+0000","last_undegraded":"2024-09-04T21:55:08.493669+0000","last_fullsized":"2024-09-04T21:55:08.493669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:52:02.322717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050858999999999995,"stat_sum":{"num_bytes":55839346,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":5,"num_read":3449,"num_read_kb":12962,"num_write":623,"num_write_kb":5453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'232","reported_seq":1149,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.613697+0000","last_change":"2024-09-04T21:54:51.359954+0000","last_active":"2024-09-04T21:55:07.613697+0000","last_peered":"2024-09-04T21:55:07.613697+0000","last_clean":"2024-09-04T21:55:07.613697+0000","last_became_active":"2024-09-04T21:54:51.359361+0000","last_became_peered":"2024-09-04T21:54:51.359361+0000","last_unstale":"2024-09-04T21:55:07.613697+0000","last_undegraded":"2024-09-04T21:55:07.613697+0000","last_fullsized":"2024-09-04T21:55:07.613697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:04:42.858136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024267529999999998,"stat_sum":{"num_bytes":4155919,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":5,"num_read":2048,"num_read_kb":4981,"num_write":302,"num_write_kb":4462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1790020,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"77'213","reported_seq":2021,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612949+0000","last_change":"2024-09-04T21:54:42.680365+0000","last_active":"2024-09-04T21:55:07.612949+0000","last_peered":"2024-09-04T21:55:07.612949+0000","last_clean":"2024-09-04T21:55:07.612949+0000","last_became_active":"2024-09-04T21:54:42.679512+0000","last_became_peered":"2024-09-04T21:54:42.679512+0000","last_unstale":"2024-09-04T21:55:07.612949+0000","last_undegraded":"2024-09-04T21:55:07.612949+0000","last_fullsized":"2024-09-04T21:55:07.612949+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:50:02.898661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361924,"stat_sum":{"num_bytes":35698199,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":10,"num_read":2008,"num_read_kb":11783,"num_write":215,"num_write_kb":5603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"83'232","reported_seq":9994,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.601244+0000","last_change":"2024-09-04T21:54:31.623898+0000","last_active":"2024-09-04T21:55:09.601244+0000","last_peered":"2024-09-04T21:55:09.601244+0000","last_clean":"2024-09-04T21:55:09.601244+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:55:09.601244+0000","last_undegraded":"2024-09-04T21:55:09.601244+0000","last_fullsized":"2024-09-04T21:55:09.601244+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:53.289644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091741999999999995,"stat_sum":{"num_bytes":39855449,"num_objects":32,"num_object_clones":11,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":11,"num_read":10477,"num_read_kb":16078,"num_write":257,"num_write_kb":4533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"77'169","reported_seq":641,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499809+0000","last_change":"2024-09-04T21:54:31.719167+0000","last_active":"2024-09-04T21:55:09.499809+0000","last_peered":"2024-09-04T21:55:09.499809+0000","last_clean":"2024-09-04T21:55:09.499809+0000","last_became_active":"2024-09-04T21:54:31.584135+0000","last_became_peered":"2024-09-04T21:54:31.584135+0000","last_unstale":"2024-09-04T21:55:09.499809+0000","last_undegraded":"2024-09-04T21:55:09.499809+0000","last_fullsized":"2024-09-04T21:55:09.499809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:40:49.570349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066328299999999997,"stat_sum":{"num_bytes":14867230,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":1552,"num_read_kb":11500,"num_write":217,"num_write_kb":4751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":894295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"82'297","reported_seq":10095,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.601682+0000","last_change":"2024-09-04T21:54:34.612521+0000","last_active":"2024-09-04T21:55:09.601682+0000","last_peered":"2024-09-04T21:55:09.601682+0000","last_clean":"2024-09-04T21:55:09.601682+0000","last_became_active":"2024-09-04T21:54:34.608209+0000","last_became_peered":"2024-09-04T21:54:34.608209+0000","last_unstale":"2024-09-04T21:55:09.601682+0000","last_undegraded":"2024-09-04T21:55:09.601682+0000","last_fullsized":"2024-09-04T21:55:09.601682+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:29:47.308039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025152799999999999,"stat_sum":{"num_bytes":50673889,"num_objects":31,"num_object_clones":10,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":9,"num_read":10240,"num_read_kb":10160,"num_write":336,"num_write_kb":9431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":717304,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"77'539","reported_seq":2494,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493179+0000","last_change":"2024-09-04T21:54:51.359507+0000","last_active":"2024-09-04T21:55:08.493179+0000","last_peered":"2024-09-04T21:55:08.493179+0000","last_clean":"2024-09-04T21:55:08.493179+0000","last_became_active":"2024-09-04T21:54:51.358608+0000","last_became_peered":"2024-09-04T21:54:51.358608+0000","last_unstale":"2024-09-04T21:55:08.493179+0000","last_undegraded":"2024-09-04T21:55:08.493179+0000","last_fullsized":"2024-09-04T21:55:08.493179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:24:39.330687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000478915,"stat_sum":{"num_bytes":9277219,"num_objects":45,"num_object_clones":12,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":15,"num_read":4257,"num_read_kb":10725,"num_write":724,"num_write_kb":7691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3966726,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'430","reported_seq":5280,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499451+0000","last_change":"2024-09-04T21:54:34.612100+0000","last_active":"2024-09-04T21:55:09.499451+0000","last_peered":"2024-09-04T21:55:09.499451+0000","last_clean":"2024-09-04T21:55:09.499451+0000","last_became_active":"2024-09-04T21:54:34.607984+0000","last_became_peered":"2024-09-04T21:54:34.607984+0000","last_unstale":"2024-09-04T21:55:09.499451+0000","last_undegraded":"2024-09-04T21:55:09.499451+0000","last_fullsized":"2024-09-04T21:55:09.499451+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:59:30.074763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023455609999999999,"stat_sum":{"num_bytes":34343653,"num_objects":35,"num_object_clones":10,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":10,"num_read":6420,"num_read_kb":41981,"num_write":580,"num_write_kb":24675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":6482454,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"77'496","reported_seq":12539,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.601734+0000","last_change":"2024-09-04T21:54:51.350972+0000","last_active":"2024-09-04T21:55:09.601734+0000","last_peered":"2024-09-04T21:55:09.601734+0000","last_clean":"2024-09-04T21:55:09.601734+0000","last_became_active":"2024-09-04T21:54:51.349872+0000","last_became_peered":"2024-09-04T21:54:51.349872+0000","last_unstale":"2024-09-04T21:55:09.601734+0000","last_undegraded":"2024-09-04T21:55:09.601734+0000","last_fullsized":"2024-09-04T21:55:09.601734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:06:19.249352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010422490000000001,"stat_sum":{"num_bytes":61410620,"num_objects":48,"num_object_clones":18,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":10,"num_read":13592,"num_read_kb":36901,"num_write":652,"num_write_kb":8893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":7361343,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'400","reported_seq":1444,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499509+0000","last_change":"2024-09-04T21:54:34.609339+0000","last_active":"2024-09-04T21:55:09.499509+0000","last_peered":"2024-09-04T21:55:09.499509+0000","last_clean":"2024-09-04T21:55:09.499509+0000","last_became_active":"2024-09-04T21:54:34.607781+0000","last_became_peered":"2024-09-04T21:54:34.607781+0000","last_unstale":"2024-09-04T21:55:09.499509+0000","last_undegraded":"2024-09-04T21:55:09.499509+0000","last_fullsized":"2024-09-04T21:55:09.499509+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:21:58.659271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091209699999999995,"stat_sum":{"num_bytes":20037035,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":8,"num_read":2098,"num_read_kb":14981,"num_write":600,"num_write_kb":4970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1583654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":803023008,"num_objects":1016,"num_object_clones":290,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":293,"num_read":141070,"num_read_kb":508363,"num_write":13895,"num_write_kb":235617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":54475810,"num_keys_recovered":164,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41037824,"data_stored":345548983,"data_compressed":16716771,"data_compressed_allocated":31059968,"data_compressed_original":339972535,"omap_allocated":0,"internal_metadata":0},"log_size":10506,"ondisk_log_size":10506,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":18583,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509462,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45124,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94326716,"statfs":{"total":96636764160,"available":96590557184,"internally_reserved":0,"allocated":5730304,"data_stored":36511970,"data_compressed":1128575,"data_compressed_allocated":2899968,"data_compressed_original":35224116,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61904,"kb_used_data":8512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309936,"statfs":{"total":96636764160,"available":96573374464,"internally_reserved":0,"allocated":8716288,"data_stored":65346907,"data_compressed":3410278,"data_compressed_allocated":6045696,"data_compressed_original":63886835,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53328,"kb_used_data":8252,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94318512,"statfs":{"total":96636764160,"available":96582156288,"internally_reserved":0,"allocated":8450048,"data_stored":67176930,"data_compressed":3566953,"data_compressed_allocated":6082560,"data_compressed_original":65804640,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38736,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333104,"statfs":{"total":96636764160,"available":96597098496,"internally_reserved":0,"allocated":3465216,"data_stored":23476588,"data_compressed":1169908,"data_compressed_allocated":2035712,"data_compressed_original":22736851,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44320,"kb_used_data":8008,"kb_used_omap":4,"kb_used_meta":36283,"kb_avail":94327520,"statfs":{"total":96636764160,"available":96591380480,"internally_reserved":0,"allocated":8200192,"data_stored":64636746,"data_compressed":2952870,"data_compressed_allocated":5865472,"data_compressed_original":63412478,"omap_allocated":4394,"internal_metadata":37154518},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":5576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":5709824,"data_stored":40706666,"data_compressed":2248429,"data_compressed_allocated":3788800,"data_compressed_original":39640615,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":35643695,"data_compressed":2027336,"data_compressed_allocated":3260416,"data_compressed_original":35162080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4886528,"data_stored":39039354,"data_compressed":1270618,"data_compressed_allocated":3080192,"data_compressed_original":38203347,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7516160,"data_stored":66747123,"data_compressed":3566953,"data_compressed_allocated":6082560,"data_compressed_original":65804640,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7577600,"data_stored":63063883,"data_compressed":3342922,"data_compressed_allocated":5869568,"data_compressed_original":62052864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4796416,"data_stored":40288466,"data_compressed":2248429,"data_compressed_allocated":3788800,"data_compressed_original":39640615,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4726784,"data_stored":35479812,"data_compressed":1121840,"data_compressed_allocated":2863104,"data_compressed_original":34634292,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7172096,"data_stored":63598794,"data_compressed":2946946,"data_compressed_allocated":5828608,"data_compressed_original":62839038,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:55:10.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:55:10.943 INFO:teuthology.orchestra.run.smithi112.stdout:4698 trunc from 0xeb9e52e to 0x64708c2 2024-09-04T21:55:10.944 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:10.941+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:10.968 INFO:teuthology.orchestra.run.smithi112.stdout:4699 read 0x53114a8 thru 0x532025f (0xedb8 bytes) 2024-09-04T21:55:10.973 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:10.973 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:10.986 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:10.989 INFO:teuthology.orchestra.run.smithi112.stdout:4700 writesame 0xc861910 thru 0xc863c34 data_size 0x1(0x2325 bytes) 2024-09-04T21:55:10.989 INFO:teuthology.orchestra.run.smithi112.stdout:4702 read 0xa04d9fc thru 0xa0547a3 (0x6da8 bytes) 2024-09-04T21:55:10.989 INFO:teuthology.orchestra.run.smithi112.stdout:4704 write 0x9b1ec31 thru 0x9b2b75f (0xcb2f bytes) 2024-09-04T21:55:10.994 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:10.994 INFO:teuthology.orchestra.run.smithi112.stdout:4706 punch from 0x401a955 to 0x402183a, (0x6ee5 bytes) 2024-09-04T21:55:10.994 INFO:teuthology.orchestra.run.smithi112.stdout:4707 read 0x4e78387 thru 0x4e7fe8a (0x7b04 bytes) 2024-09-04T21:55:10.995 INFO:teuthology.orchestra.run.smithi112.stdout:4708 writesame 0xb7f72a9 thru 0xb80418c data_size 0x2(0xcee4 bytes) 2024-09-04T21:55:11.009 INFO:teuthology.orchestra.run.smithi112.stdout:4709 writesame 0xa80216e thru 0xa80aaf4 data_size 0x13(0x8987 bytes) 2024-09-04T21:55:11.016 INFO:teuthology.orchestra.run.smithi112.stdout:4710 write 0x426e12 thru 0x4361c3 (0xf3b2 bytes) 2024-09-04T21:55:11.022 INFO:teuthology.orchestra.run.smithi112.stdout:4711 writesame 0x5cc2387 thru 0x5cd08bd data_size 0x1(0xe537 bytes) 2024-09-04T21:55:11.045 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:11.045+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:11.062 INFO:teuthology.orchestra.run.smithi112.stdout:4712 write 0xba59076 thru 0xba5af90 (0x1f1b bytes) 2024-09-04T21:55:11.066 INFO:teuthology.orchestra.run.smithi112.stdout:4713 write 0x9e8c743 thru 0x9e9b432 (0xecf0 bytes) 2024-09-04T21:55:11.070 INFO:teuthology.orchestra.run.smithi112.stdout:4714 writesame 0x28472a9 thru 0x2851deb data_size 0x1(0xab43 bytes) 2024-09-04T21:55:11.094 INFO:teuthology.orchestra.run.smithi112.stdout:4716 writesame 0xae2cb55 thru 0xae2de22 data_size 0x1d(0x12ce bytes) 2024-09-04T21:55:11.098 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:11.098 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:11.098 INFO:teuthology.orchestra.run.smithi112.stdout:4720 write 0x357ebd7 thru 0x358c83e (0xdc68 bytes) 2024-09-04T21:55:11.104 INFO:teuthology.orchestra.run.smithi112.stdout:4721 write 0x82fd680 thru 0x8305d28 (0x86a9 bytes) 2024-09-04T21:55:11.121 INFO:teuthology.orchestra.run.smithi112.stdout:4722 trunc from 0xc863c35 to 0x4e1d280 2024-09-04T21:55:11.142 INFO:teuthology.orchestra.run.smithi112.stdout:4723 read 0x2e1fa54 thru 0x2e27f63 (0x8510 bytes) 2024-09-04T21:55:11.142 INFO:teuthology.orchestra.run.smithi112.stdout:4724 read 0x2ecc17e thru 0x2ed1e0d (0x5c90 bytes) 2024-09-04T21:55:11.142 INFO:teuthology.orchestra.run.smithi112.stdout:4725 punch from 0x39d0e2e to 0x39d27c6, (0x1998 bytes) 2024-09-04T21:55:11.142 INFO:teuthology.orchestra.run.smithi112.stdout:4726 read 0x495f4d7 thru 0x496c4d8 (0xd002 bytes) 2024-09-04T21:55:11.142 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:11.142 INFO:teuthology.orchestra.run.smithi112.stdout:4728 read 0x43ce130 thru 0x43d3d00 (0x5bd1 bytes) 2024-09-04T21:55:11.145 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:11.145+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:11.157 INFO:teuthology.orchestra.run.smithi112.stdout:4730 write 0x7009267 thru 0x70156d4 (0xc46e bytes) 2024-09-04T21:55:11.158 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:11.158 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:11.163 INFO:teuthology.orchestra.run.smithi112.stdout:4731 read 0x19ef5ee thru 0x19f82e0 (0x8cf3 bytes) 2024-09-04T21:55:11.164 INFO:teuthology.orchestra.run.smithi112.stdout:4733 read 0x1670d53 thru 0x167bc4d (0xaefb bytes) 2024-09-04T21:55:11.165 INFO:teuthology.orchestra.run.smithi112.stdout:4735 read 0x4c7fa11 thru 0x4c8d97e (0xdf6e bytes) 2024-09-04T21:55:11.166 INFO:teuthology.orchestra.run.smithi112.stdout:4736 read 0x1b5eb4c thru 0x1b68a49 (0x9efe bytes) 2024-09-04T21:55:11.167 INFO:teuthology.orchestra.run.smithi112.stdout:4737 read 0x6fa5d60 thru 0x6fb1860 (0xbb01 bytes) 2024-09-04T21:55:11.172 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:11.177 INFO:teuthology.orchestra.run.smithi112.stdout:4738 trunc from 0x70156d5 to 0x23c0c48 2024-09-04T21:55:11.216 INFO:teuthology.orchestra.run.smithi112.stdout:4739 write 0x958fc41 thru 0x9592476 (0x2836 bytes) 2024-09-04T21:55:11.220 INFO:teuthology.orchestra.run.smithi112.stdout:4740 read 0x4e072e5 thru 0x4e0d06e (0x5d8a bytes) 2024-09-04T21:55:11.232 INFO:teuthology.orchestra.run.smithi112.stdout:4741 writesame 0xb2d23ab thru 0xb2d8351 data_size 0x1(0x5fa7 bytes) 2024-09-04T21:55:11.233 INFO:teuthology.orchestra.run.smithi112.stdout:4742 read 0x405b192 thru 0x40638cd (0x873c bytes) 2024-09-04T21:55:11.233 INFO:teuthology.orchestra.run.smithi112.stdout:4743 writesame 0x36d1f3e thru 0x36d40cc data_size 0xb(0x218f bytes) 2024-09-04T21:55:11.237 INFO:teuthology.orchestra.run.smithi112.stdout:4744 read 0x12ef667 thru 0x12f1204 (0x1b9e bytes) 2024-09-04T21:55:11.238 INFO:teuthology.orchestra.run.smithi112.stdout:4745 write 0x13c342 thru 0x13d66d (0x132c bytes) 2024-09-04T21:55:11.245 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:11.245+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:11.247 INFO:teuthology.orchestra.run.smithi112.stdout:4746 write 0x82cc038 thru 0x82d695a (0xa923 bytes) 2024-09-04T21:55:11.254 INFO:teuthology.orchestra.run.smithi112.stdout:4747 punch from 0x96c4320 to 0x96cfd6e, (0xba4e bytes) 2024-09-04T21:55:11.254 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:11.254 INFO:teuthology.orchestra.run.smithi112.stdout:4749 punch from 0x1995643 to 0x199d749, (0x8106 bytes) 2024-09-04T21:55:11.254 INFO:teuthology.orchestra.run.smithi112.stdout:4750 writesame 0x3624d77 thru 0x36286b5 data_size 0x5(0x393f bytes) 2024-09-04T21:55:11.259 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:55:11.260 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:55:11.261 INFO:teuthology.orchestra.run.smithi112.stdout:4751 writesame 0x5adc4b thru 0x5b1920 data_size 0x2(0x3cd6 bytes) 2024-09-04T21:55:11.279 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-04T21:55:10.496224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":803613395,"num_objects":1020,"num_object_clones":290,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":293,"num_read":141136,"num_read_kb":508418,"num_write":13998,"num_write_kb":237247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":55066178,"num_keys_recovered":164,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10581,"ondisk_log_size":10581,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":276016,"kb_used_data":39328,"kb_used_omap":12,"kb_used_meta":236531,"kb_avail":565955024,"statfs":{"total":579820584960,"available":579537944576,"internally_reserved":0,"allocated":40271872,"data_stored":297855807,"data_compressed":14477013,"data_compressed_allocated":26718208,"data_compressed_original":290705535,"omap_allocated":12366,"internal_metadata":242208690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47428803,"num_objects":83,"num_object_clones":42,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":47,"num_read":24211,"num_read_kb":28549,"num_write":620,"num_write_kb":24491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030567"},"pg_stats":[{"pgid":"3.1f","version":"77'170","reported_seq":814,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485966+0000","last_change":"2024-09-04T21:54:51.351220+0000","last_active":"2024-09-04T21:55:07.485966+0000","last_peered":"2024-09-04T21:55:07.485966+0000","last_clean":"2024-09-04T21:55:07.485966+0000","last_became_active":"2024-09-04T21:54:51.350505+0000","last_became_peered":"2024-09-04T21:54:51.350505+0000","last_unstale":"2024-09-04T21:55:07.485966+0000","last_undegraded":"2024-09-04T21:55:07.485966+0000","last_fullsized":"2024-09-04T21:55:07.485966+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:48:33.984938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003772475,"stat_sum":{"num_bytes":15729141,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":9,"num_read":985,"num_read_kb":2424,"num_write":181,"num_write_kb":2132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":330234,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"81'208","reported_seq":1948,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.701793+0000","last_change":"2024-09-04T21:53:57.370393+0000","last_active":"2024-09-04T21:55:07.701793+0000","last_peered":"2024-09-04T21:55:07.701793+0000","last_clean":"2024-09-04T21:55:07.701793+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:55:07.701793+0000","last_undegraded":"2024-09-04T21:55:07.701793+0000","last_fullsized":"2024-09-04T21:55:07.701793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:07:09.045775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372790000000002,"stat_sum":{"num_bytes":16454396,"num_objects":32,"num_object_clones":8,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":8,"num_read":1758,"num_read_kb":8942,"num_write":202,"num_write_kb":4987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'405","reported_seq":14103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487294+0000","last_change":"2024-09-04T21:54:42.681915+0000","last_active":"2024-09-04T21:55:08.487294+0000","last_peered":"2024-09-04T21:55:08.487294+0000","last_clean":"2024-09-04T21:55:08.487294+0000","last_became_active":"2024-09-04T21:54:42.681042+0000","last_became_peered":"2024-09-04T21:54:42.681042+0000","last_unstale":"2024-09-04T21:55:08.487294+0000","last_undegraded":"2024-09-04T21:55:08.487294+0000","last_fullsized":"2024-09-04T21:55:08.487294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:47:35.449231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00153718,"stat_sum":{"num_bytes":70048262,"num_objects":33,"num_object_clones":11,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":7,"num_read":14914,"num_read_kb":17060,"num_write":606,"num_write_kb":8024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"82'235","reported_seq":1946,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.722400+0000","last_change":"2024-09-04T21:54:42.680392+0000","last_active":"2024-09-04T21:55:08.722400+0000","last_peered":"2024-09-04T21:55:08.722400+0000","last_clean":"2024-09-04T21:55:08.722400+0000","last_became_active":"2024-09-04T21:54:42.679589+0000","last_became_peered":"2024-09-04T21:54:42.679589+0000","last_unstale":"2024-09-04T21:55:08.722400+0000","last_undegraded":"2024-09-04T21:55:08.722400+0000","last_fullsized":"2024-09-04T21:55:08.722400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:39:37.588120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349371,"stat_sum":{"num_bytes":21274371,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":1679,"num_read_kb":20468,"num_write":235,"num_write_kb":4710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"81'171","reported_seq":1339,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487235+0000","last_change":"2024-09-04T21:54:51.355968+0000","last_active":"2024-09-04T21:55:08.487235+0000","last_peered":"2024-09-04T21:55:08.487235+0000","last_clean":"2024-09-04T21:55:08.487235+0000","last_became_active":"2024-09-04T21:54:51.354934+0000","last_became_peered":"2024-09-04T21:54:51.354934+0000","last_unstale":"2024-09-04T21:55:08.487235+0000","last_undegraded":"2024-09-04T21:55:08.487235+0000","last_fullsized":"2024-09-04T21:55:08.487235+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:41:34.673551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295954,"stat_sum":{"num_bytes":5986096,"num_objects":22,"num_object_clones":8,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":6,"num_read":967,"num_read_kb":5764,"num_write":169,"num_write_kb":5902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2174559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"77'630","reported_seq":11557,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.718823+0000","last_change":"2024-09-04T21:54:13.554263+0000","last_active":"2024-09-04T21:55:08.718823+0000","last_peered":"2024-09-04T21:55:08.718823+0000","last_clean":"2024-09-04T21:55:08.718823+0000","last_became_active":"2024-09-04T21:54:13.481244+0000","last_became_peered":"2024-09-04T21:54:13.481244+0000","last_unstale":"2024-09-04T21:55:08.718823+0000","last_undegraded":"2024-09-04T21:55:08.718823+0000","last_fullsized":"2024-09-04T21:55:08.718823+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:56:40.949724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001385184,"stat_sum":{"num_bytes":33356472,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":11,"num_read":13125,"num_read_kb":42576,"num_write":946,"num_write_kb":8663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":930020,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"81'1063","reported_seq":1987,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.624734+0000","last_change":"2024-09-04T21:54:13.510809+0000","last_active":"2024-09-04T21:55:07.624734+0000","last_peered":"2024-09-04T21:55:07.624734+0000","last_clean":"2024-09-04T21:55:07.624734+0000","last_became_active":"2024-09-04T21:54:13.478356+0000","last_became_peered":"2024-09-04T21:54:13.478356+0000","last_unstale":"2024-09-04T21:55:07.624734+0000","last_undegraded":"2024-09-04T21:55:07.624734+0000","last_fullsized":"2024-09-04T21:55:07.624734+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1063,"log_dups_size":0,"ondisk_log_size":1063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:10:19.924484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032359149999999998,"stat_sum":{"num_bytes":14896953,"num_objects":29,"num_object_clones":8,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":10,"num_read":5877,"num_read_kb":15599,"num_write":1782,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4393,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"77'263","reported_seq":3130,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612791+0000","last_change":"2024-09-04T21:54:51.350327+0000","last_active":"2024-09-04T21:55:07.612791+0000","last_peered":"2024-09-04T21:55:07.612791+0000","last_clean":"2024-09-04T21:55:07.612791+0000","last_became_active":"2024-09-04T21:54:51.349573+0000","last_became_peered":"2024-09-04T21:54:51.349573+0000","last_unstale":"2024-09-04T21:55:07.612791+0000","last_undegraded":"2024-09-04T21:55:07.612791+0000","last_fullsized":"2024-09-04T21:55:07.612791+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:49:24.297704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001549499,"stat_sum":{"num_bytes":23194872,"num_objects":25,"num_object_clones":8,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":8,"num_read":4048,"num_read_kb":24063,"num_write":285,"num_write_kb":7248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3632211,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"77'767","reported_seq":2379,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.613041+0000","last_change":"2024-09-04T21:54:51.353124+0000","last_active":"2024-09-04T21:55:07.613041+0000","last_peered":"2024-09-04T21:55:07.613041+0000","last_clean":"2024-09-04T21:55:07.613041+0000","last_became_active":"2024-09-04T21:54:51.352376+0000","last_became_peered":"2024-09-04T21:54:51.352376+0000","last_unstale":"2024-09-04T21:55:07.613041+0000","last_undegraded":"2024-09-04T21:55:07.613041+0000","last_fullsized":"2024-09-04T21:55:07.613041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:41:22.321182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035280929999999999,"stat_sum":{"num_bytes":31613458,"num_objects":37,"num_object_clones":4,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":13,"num_read":4973,"num_read_kb":21787,"num_write":1160,"num_write_kb":6773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3732324,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'123","reported_seq":1314,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499711+0000","last_change":"2024-09-04T21:54:42.680486+0000","last_active":"2024-09-04T21:55:09.499711+0000","last_peered":"2024-09-04T21:55:09.499711+0000","last_clean":"2024-09-04T21:55:09.499711+0000","last_became_active":"2024-09-04T21:54:42.679389+0000","last_became_peered":"2024-09-04T21:54:42.679389+0000","last_unstale":"2024-09-04T21:55:09.499711+0000","last_undegraded":"2024-09-04T21:55:09.499711+0000","last_fullsized":"2024-09-04T21:55:09.499711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:49:40.549041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086423499999999998,"stat_sum":{"num_bytes":13680299,"num_objects":18,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":1016,"num_read_kb":13715,"num_write":130,"num_write_kb":4823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"77'200","reported_seq":1748,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499897+0000","last_change":"2024-09-04T21:54:31.683707+0000","last_active":"2024-09-04T21:55:09.499897+0000","last_peered":"2024-09-04T21:55:09.499897+0000","last_clean":"2024-09-04T21:55:09.499897+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:55:09.499897+0000","last_undegraded":"2024-09-04T21:55:09.499897+0000","last_fullsized":"2024-09-04T21:55:09.499897+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:29:12.336532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078592199999999999,"stat_sum":{"num_bytes":18886378,"num_objects":40,"num_object_clones":11,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":18,"num_read":1900,"num_read_kb":19234,"num_write":199,"num_write_kb":14334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"77'162","reported_seq":1379,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487788+0000","last_change":"2024-09-04T21:54:31.673826+0000","last_active":"2024-09-04T21:55:08.487788+0000","last_peered":"2024-09-04T21:55:08.487788+0000","last_clean":"2024-09-04T21:55:08.487788+0000","last_became_active":"2024-09-04T21:54:31.582403+0000","last_became_peered":"2024-09-04T21:54:31.582403+0000","last_unstale":"2024-09-04T21:55:08.487788+0000","last_undegraded":"2024-09-04T21:55:08.487788+0000","last_fullsized":"2024-09-04T21:55:08.487788+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:36:22.287956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001035425,"stat_sum":{"num_bytes":9043312,"num_objects":30,"num_object_clones":11,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":1141,"num_read_kb":5869,"num_write":188,"num_write_kb":3810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":529142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493142+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:55:08.493142+0000","last_peered":"2024-09-04T21:55:08.493142+0000","last_clean":"2024-09-04T21:55:08.493142+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:55:08.493142+0000","last_undegraded":"2024-09-04T21:55:08.493142+0000","last_fullsized":"2024-09-04T21:55:08.493142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"77'210","reported_seq":962,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.488090+0000","last_change":"2024-09-04T21:54:51.354834+0000","last_active":"2024-09-04T21:55:08.488090+0000","last_peered":"2024-09-04T21:55:08.488090+0000","last_clean":"2024-09-04T21:55:08.488090+0000","last_became_active":"2024-09-04T21:54:51.353742+0000","last_became_peered":"2024-09-04T21:54:51.353742+0000","last_unstale":"2024-09-04T21:55:08.488090+0000","last_undegraded":"2024-09-04T21:55:08.488090+0000","last_fullsized":"2024-09-04T21:55:08.488090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:21:20.454105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001390173,"stat_sum":{"num_bytes":5219982,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":6,"num_read":862,"num_read_kb":2160,"num_write":205,"num_write_kb":2877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":645779,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493295+0000","last_change":"2024-09-04T21:54:13.481230+0000","last_active":"2024-09-04T21:55:08.493295+0000","last_peered":"2024-09-04T21:55:08.493295+0000","last_clean":"2024-09-04T21:55:08.493295+0000","last_became_active":"2024-09-04T21:54:13.479390+0000","last_became_peered":"2024-09-04T21:54:13.479390+0000","last_unstale":"2024-09-04T21:55:08.493295+0000","last_undegraded":"2024-09-04T21:55:08.493295+0000","last_fullsized":"2024-09-04T21:55:08.493295+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:05.478515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"77'173","reported_seq":2608,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.613090+0000","last_change":"2024-09-04T21:54:31.773042+0000","last_active":"2024-09-04T21:55:07.613090+0000","last_peered":"2024-09-04T21:55:07.613090+0000","last_clean":"2024-09-04T21:55:07.613090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:55:07.613090+0000","last_undegraded":"2024-09-04T21:55:07.613090+0000","last_fullsized":"2024-09-04T21:55:07.613090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:32:32.369703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015346380000000001,"stat_sum":{"num_bytes":10882634,"num_objects":37,"num_object_clones":12,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":9,"num_read":2848,"num_read_kb":19568,"num_write":226,"num_write_kb":9463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499950+0000","last_change":"2024-09-04T21:54:48.374647+0000","last_active":"2024-09-04T21:55:09.499950+0000","last_peered":"2024-09-04T21:55:09.499950+0000","last_clean":"2024-09-04T21:55:09.499950+0000","last_became_active":"2024-09-04T21:54:48.374288+0000","last_became_peered":"2024-09-04T21:54:48.374288+0000","last_unstale":"2024-09-04T21:55:09.499950+0000","last_undegraded":"2024-09-04T21:55:09.499950+0000","last_fullsized":"2024-09-04T21:55:09.499950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:14:06.038769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"77'187","reported_seq":1247,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485820+0000","last_change":"2024-09-04T21:54:13.539266+0000","last_active":"2024-09-04T21:55:07.485820+0000","last_peered":"2024-09-04T21:55:07.485820+0000","last_clean":"2024-09-04T21:55:07.485820+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:55:07.485820+0000","last_undegraded":"2024-09-04T21:55:07.485820+0000","last_fullsized":"2024-09-04T21:55:07.485820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:30:25.920556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091729799999999998,"stat_sum":{"num_bytes":7414563,"num_objects":22,"num_object_clones":6,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":6,"num_read":1540,"num_read_kb":5820,"num_write":189,"num_write_kb":2889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":82,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499504+0000","last_change":"2024-09-04T21:54:48.372640+0000","last_active":"2024-09-04T21:55:09.499504+0000","last_peered":"2024-09-04T21:55:09.499504+0000","last_clean":"2024-09-04T21:55:09.499504+0000","last_became_active":"2024-09-04T21:54:48.372106+0000","last_became_peered":"2024-09-04T21:54:48.372106+0000","last_unstale":"2024-09-04T21:55:09.499504+0000","last_undegraded":"2024-09-04T21:55:09.499504+0000","last_fullsized":"2024-09-04T21:55:09.499504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:18:59.210273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"77'375","reported_seq":8783,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612935+0000","last_change":"2024-09-04T21:54:42.680694+0000","last_active":"2024-09-04T21:55:07.612935+0000","last_peered":"2024-09-04T21:55:07.612935+0000","last_clean":"2024-09-04T21:55:07.612935+0000","last_became_active":"2024-09-04T21:54:42.679683+0000","last_became_peered":"2024-09-04T21:54:42.679683+0000","last_unstale":"2024-09-04T21:55:07.612935+0000","last_undegraded":"2024-09-04T21:55:07.612935+0000","last_fullsized":"2024-09-04T21:55:07.612935+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:00:31.225665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012888800000000001,"stat_sum":{"num_bytes":29219111,"num_objects":28,"num_object_clones":9,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":6,"num_read":9766,"num_read_kb":22284,"num_write":454,"num_write_kb":9513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"77'195","reported_seq":2842,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487886+0000","last_change":"2024-09-04T21:54:51.357970+0000","last_active":"2024-09-04T21:55:08.487886+0000","last_peered":"2024-09-04T21:55:08.487886+0000","last_clean":"2024-09-04T21:55:08.487886+0000","last_became_active":"2024-09-04T21:54:51.357182+0000","last_became_peered":"2024-09-04T21:54:51.357182+0000","last_unstale":"2024-09-04T21:55:08.487886+0000","last_undegraded":"2024-09-04T21:55:08.487886+0000","last_fullsized":"2024-09-04T21:55:08.487886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:28:43.729296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050067700000000004,"stat_sum":{"num_bytes":13631355,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":2488,"num_read_kb":14671,"num_write":215,"num_write_kb":11174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":2060586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":103,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.492929+0000","last_change":"2024-09-04T21:54:42.675259+0000","last_active":"2024-09-04T21:55:08.492929+0000","last_peered":"2024-09-04T21:55:08.492929+0000","last_clean":"2024-09-04T21:55:08.492929+0000","last_became_active":"2024-09-04T21:54:42.674803+0000","last_became_peered":"2024-09-04T21:54:42.674803+0000","last_unstale":"2024-09-04T21:55:08.492929+0000","last_undegraded":"2024-09-04T21:55:08.492929+0000","last_fullsized":"2024-09-04T21:55:08.492929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:09:21.260817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'250","reported_seq":2109,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485329+0000","last_change":"2024-09-04T21:54:16.342878+0000","last_active":"2024-09-04T21:55:07.485329+0000","last_peered":"2024-09-04T21:55:07.485329+0000","last_clean":"2024-09-04T21:55:07.485329+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:55:07.485329+0000","last_undegraded":"2024-09-04T21:55:07.485329+0000","last_fullsized":"2024-09-04T21:55:07.485329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:51:50.337051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010876340000000001,"stat_sum":{"num_bytes":31643640,"num_objects":43,"num_object_clones":11,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":2751,"num_read_kb":9058,"num_write":490,"num_write_kb":20359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485467+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:55:07.485467+0000","last_peered":"2024-09-04T21:55:07.485467+0000","last_clean":"2024-09-04T21:55:07.485467+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:55:07.485467+0000","last_undegraded":"2024-09-04T21:55:07.485467+0000","last_fullsized":"2024-09-04T21:55:07.485467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"76'344","reported_seq":2836,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.564723+0000","last_change":"2024-09-04T21:54:51.354264+0000","last_active":"2024-09-04T21:55:08.564723+0000","last_peered":"2024-09-04T21:55:08.564723+0000","last_clean":"2024-09-04T21:55:08.564723+0000","last_became_active":"2024-09-04T21:54:51.353319+0000","last_became_peered":"2024-09-04T21:54:51.353319+0000","last_unstale":"2024-09-04T21:55:08.564723+0000","last_undegraded":"2024-09-04T21:55:08.564723+0000","last_fullsized":"2024-09-04T21:55:08.564723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:06.335488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001396415,"stat_sum":{"num_bytes":42089206,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":3633,"num_read_kb":25388,"num_write":498,"num_write_kb":5581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3580019,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499453+0000","last_change":"2024-09-04T21:54:48.373076+0000","last_active":"2024-09-04T21:55:09.499453+0000","last_peered":"2024-09-04T21:55:09.499453+0000","last_clean":"2024-09-04T21:55:09.499453+0000","last_became_active":"2024-09-04T21:54:48.372613+0000","last_became_peered":"2024-09-04T21:54:48.372613+0000","last_unstale":"2024-09-04T21:55:09.499453+0000","last_undegraded":"2024-09-04T21:55:09.499453+0000","last_fullsized":"2024-09-04T21:55:09.499453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:48:13.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"77'284","reported_seq":2738,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612834+0000","last_change":"2024-09-04T21:53:57.369713+0000","last_active":"2024-09-04T21:55:07.612834+0000","last_peered":"2024-09-04T21:55:07.612834+0000","last_clean":"2024-09-04T21:55:07.612834+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:55:07.612834+0000","last_undegraded":"2024-09-04T21:55:07.612834+0000","last_fullsized":"2024-09-04T21:55:07.612834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:28.880126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026956770000000001,"stat_sum":{"num_bytes":27973372,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":3388,"num_read_kb":13649,"num_write":277,"num_write_kb":4740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.485604+0000","last_change":"2024-09-04T21:54:42.676579+0000","last_active":"2024-09-04T21:55:07.485604+0000","last_peered":"2024-09-04T21:55:07.485604+0000","last_clean":"2024-09-04T21:55:07.485604+0000","last_became_active":"2024-09-04T21:54:42.676301+0000","last_became_peered":"2024-09-04T21:54:42.676301+0000","last_unstale":"2024-09-04T21:55:07.485604+0000","last_undegraded":"2024-09-04T21:55:07.485604+0000","last_fullsized":"2024-09-04T21:55:07.485604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:42:21.618311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":192,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499421+0000","last_change":"2024-09-04T21:54:48.383973+0000","last_active":"2024-09-04T21:55:09.499421+0000","last_peered":"2024-09-04T21:55:09.499421+0000","last_clean":"2024-09-04T21:55:09.499421+0000","last_became_active":"2024-09-04T21:54:48.374571+0000","last_became_peered":"2024-09-04T21:54:48.374571+0000","last_unstale":"2024-09-04T21:55:09.499421+0000","last_undegraded":"2024-09-04T21:55:09.499421+0000","last_fullsized":"2024-09-04T21:55:09.499421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:54:44.340649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"77'427","reported_seq":2724,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612829+0000","last_change":"2024-09-04T21:54:34.607151+0000","last_active":"2024-09-04T21:55:07.612829+0000","last_peered":"2024-09-04T21:55:07.612829+0000","last_clean":"2024-09-04T21:55:07.612829+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:55:07.612829+0000","last_undegraded":"2024-09-04T21:55:07.612829+0000","last_fullsized":"2024-09-04T21:55:07.612829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:46:29.393261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001397155,"stat_sum":{"num_bytes":20789094,"num_objects":32,"num_object_clones":10,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":6,"num_read":3848,"num_read_kb":13427,"num_write":567,"num_write_kb":6154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"77'196","reported_seq":1730,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.487255+0000","last_change":"2024-09-04T21:53:57.346105+0000","last_active":"2024-09-04T21:55:08.487255+0000","last_peered":"2024-09-04T21:55:08.487255+0000","last_clean":"2024-09-04T21:55:08.487255+0000","last_became_active":"2024-09-04T21:53:55.345842+0000","last_became_peered":"2024-09-04T21:53:55.345842+0000","last_unstale":"2024-09-04T21:55:08.487255+0000","last_undegraded":"2024-09-04T21:55:08.487255+0000","last_fullsized":"2024-09-04T21:55:08.487255+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:40:38.827361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094585800000000005,"stat_sum":{"num_bytes":13837482,"num_objects":35,"num_object_clones":13,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":13,"num_read":1422,"num_read_kb":12785,"num_write":185,"num_write_kb":5335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"82'460","reported_seq":2175,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493669+0000","last_change":"2024-09-04T21:53:57.339901+0000","last_active":"2024-09-04T21:55:08.493669+0000","last_peered":"2024-09-04T21:55:08.493669+0000","last_clean":"2024-09-04T21:55:08.493669+0000","last_became_active":"2024-09-04T21:53:55.355393+0000","last_became_peered":"2024-09-04T21:53:55.355393+0000","last_unstale":"2024-09-04T21:55:08.493669+0000","last_undegraded":"2024-09-04T21:55:08.493669+0000","last_fullsized":"2024-09-04T21:55:08.493669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:52:02.322717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050858999999999995,"stat_sum":{"num_bytes":55839346,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":5,"num_read":3449,"num_read_kb":12962,"num_write":623,"num_write_kb":5453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'232","reported_seq":1149,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.613697+0000","last_change":"2024-09-04T21:54:51.359954+0000","last_active":"2024-09-04T21:55:07.613697+0000","last_peered":"2024-09-04T21:55:07.613697+0000","last_clean":"2024-09-04T21:55:07.613697+0000","last_became_active":"2024-09-04T21:54:51.359361+0000","last_became_peered":"2024-09-04T21:54:51.359361+0000","last_unstale":"2024-09-04T21:55:07.613697+0000","last_undegraded":"2024-09-04T21:55:07.613697+0000","last_fullsized":"2024-09-04T21:55:07.613697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:04:42.858136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024267529999999998,"stat_sum":{"num_bytes":4155919,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":5,"num_read":2048,"num_read_kb":4981,"num_write":302,"num_write_kb":4462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1790020,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"77'213","reported_seq":2021,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-04T21:55:07.612949+0000","last_change":"2024-09-04T21:54:42.680365+0000","last_active":"2024-09-04T21:55:07.612949+0000","last_peered":"2024-09-04T21:55:07.612949+0000","last_clean":"2024-09-04T21:55:07.612949+0000","last_became_active":"2024-09-04T21:54:42.679512+0000","last_became_peered":"2024-09-04T21:54:42.679512+0000","last_unstale":"2024-09-04T21:55:07.612949+0000","last_undegraded":"2024-09-04T21:55:07.612949+0000","last_fullsized":"2024-09-04T21:55:07.612949+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:50:02.898661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361924,"stat_sum":{"num_bytes":35698199,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":10,"num_read":2008,"num_read_kb":11783,"num_write":215,"num_write_kb":5603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"83'232","reported_seq":9994,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.601244+0000","last_change":"2024-09-04T21:54:31.623898+0000","last_active":"2024-09-04T21:55:09.601244+0000","last_peered":"2024-09-04T21:55:09.601244+0000","last_clean":"2024-09-04T21:55:09.601244+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:55:09.601244+0000","last_undegraded":"2024-09-04T21:55:09.601244+0000","last_fullsized":"2024-09-04T21:55:09.601244+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:53.289644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091741999999999995,"stat_sum":{"num_bytes":39855449,"num_objects":32,"num_object_clones":11,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":11,"num_read":10477,"num_read_kb":16078,"num_write":257,"num_write_kb":4533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"77'169","reported_seq":641,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499809+0000","last_change":"2024-09-04T21:54:31.719167+0000","last_active":"2024-09-04T21:55:09.499809+0000","last_peered":"2024-09-04T21:55:09.499809+0000","last_clean":"2024-09-04T21:55:09.499809+0000","last_became_active":"2024-09-04T21:54:31.584135+0000","last_became_peered":"2024-09-04T21:54:31.584135+0000","last_unstale":"2024-09-04T21:55:09.499809+0000","last_undegraded":"2024-09-04T21:55:09.499809+0000","last_fullsized":"2024-09-04T21:55:09.499809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:40:49.570349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066328299999999997,"stat_sum":{"num_bytes":14867230,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":1552,"num_read_kb":11500,"num_write":217,"num_write_kb":4751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":894295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"82'297","reported_seq":10095,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.601682+0000","last_change":"2024-09-04T21:54:34.612521+0000","last_active":"2024-09-04T21:55:09.601682+0000","last_peered":"2024-09-04T21:55:09.601682+0000","last_clean":"2024-09-04T21:55:09.601682+0000","last_became_active":"2024-09-04T21:54:34.608209+0000","last_became_peered":"2024-09-04T21:54:34.608209+0000","last_unstale":"2024-09-04T21:55:09.601682+0000","last_undegraded":"2024-09-04T21:55:09.601682+0000","last_fullsized":"2024-09-04T21:55:09.601682+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:29:47.308039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025152799999999999,"stat_sum":{"num_bytes":50673889,"num_objects":31,"num_object_clones":10,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":9,"num_read":10240,"num_read_kb":10160,"num_write":336,"num_write_kb":9431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":717304,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"77'539","reported_seq":2494,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-04T21:55:08.493179+0000","last_change":"2024-09-04T21:54:51.359507+0000","last_active":"2024-09-04T21:55:08.493179+0000","last_peered":"2024-09-04T21:55:08.493179+0000","last_clean":"2024-09-04T21:55:08.493179+0000","last_became_active":"2024-09-04T21:54:51.358608+0000","last_became_peered":"2024-09-04T21:54:51.358608+0000","last_unstale":"2024-09-04T21:55:08.493179+0000","last_undegraded":"2024-09-04T21:55:08.493179+0000","last_fullsized":"2024-09-04T21:55:08.493179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:24:39.330687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000478915,"stat_sum":{"num_bytes":9277219,"num_objects":45,"num_object_clones":12,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":15,"num_read":4257,"num_read_kb":10725,"num_write":724,"num_write_kb":7691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3966726,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'430","reported_seq":5280,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499451+0000","last_change":"2024-09-04T21:54:34.612100+0000","last_active":"2024-09-04T21:55:09.499451+0000","last_peered":"2024-09-04T21:55:09.499451+0000","last_clean":"2024-09-04T21:55:09.499451+0000","last_became_active":"2024-09-04T21:54:34.607984+0000","last_became_peered":"2024-09-04T21:54:34.607984+0000","last_unstale":"2024-09-04T21:55:09.499451+0000","last_undegraded":"2024-09-04T21:55:09.499451+0000","last_fullsized":"2024-09-04T21:55:09.499451+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:59:30.074763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023455609999999999,"stat_sum":{"num_bytes":34343653,"num_objects":35,"num_object_clones":10,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":10,"num_read":6420,"num_read_kb":41981,"num_write":580,"num_write_kb":24675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":6482454,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"77'496","reported_seq":12539,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.601734+0000","last_change":"2024-09-04T21:54:51.350972+0000","last_active":"2024-09-04T21:55:09.601734+0000","last_peered":"2024-09-04T21:55:09.601734+0000","last_clean":"2024-09-04T21:55:09.601734+0000","last_became_active":"2024-09-04T21:54:51.349872+0000","last_became_peered":"2024-09-04T21:54:51.349872+0000","last_unstale":"2024-09-04T21:55:09.601734+0000","last_undegraded":"2024-09-04T21:55:09.601734+0000","last_fullsized":"2024-09-04T21:55:09.601734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:06:19.249352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010422490000000001,"stat_sum":{"num_bytes":61410620,"num_objects":48,"num_object_clones":18,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":10,"num_read":13592,"num_read_kb":36901,"num_write":652,"num_write_kb":8893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":7361343,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'400","reported_seq":1444,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-04T21:55:09.499509+0000","last_change":"2024-09-04T21:54:34.609339+0000","last_active":"2024-09-04T21:55:09.499509+0000","last_peered":"2024-09-04T21:55:09.499509+0000","last_clean":"2024-09-04T21:55:09.499509+0000","last_became_active":"2024-09-04T21:54:34.607781+0000","last_became_peered":"2024-09-04T21:54:34.607781+0000","last_unstale":"2024-09-04T21:55:09.499509+0000","last_undegraded":"2024-09-04T21:55:09.499509+0000","last_fullsized":"2024-09-04T21:55:09.499509+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:21:58.659271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091209699999999995,"stat_sum":{"num_bytes":20037035,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":8,"num_read":2098,"num_read_kb":14981,"num_write":600,"num_write_kb":4970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1583654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":803023008,"num_objects":1016,"num_object_clones":290,"num_object_copies":2032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":293,"num_read":141070,"num_read_kb":508363,"num_write":13895,"num_write_kb":235617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":827,"num_bytes_recovered":54475810,"num_keys_recovered":164,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41037824,"data_stored":345548983,"data_compressed":16716771,"data_compressed_allocated":31059968,"data_compressed_original":339972535,"omap_allocated":0,"internal_metadata":0},"log_size":10506,"ondisk_log_size":10506,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":18583,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509462,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45124,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94326716,"statfs":{"total":96636764160,"available":96590557184,"internally_reserved":0,"allocated":5730304,"data_stored":36511970,"data_compressed":1128575,"data_compressed_allocated":2899968,"data_compressed_original":35224116,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61904,"kb_used_data":8512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309936,"statfs":{"total":96636764160,"available":96573374464,"internally_reserved":0,"allocated":8716288,"data_stored":65346907,"data_compressed":3410278,"data_compressed_allocated":6045696,"data_compressed_original":63886835,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53328,"kb_used_data":8252,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94318512,"statfs":{"total":96636764160,"available":96582156288,"internally_reserved":0,"allocated":8450048,"data_stored":67176930,"data_compressed":3566953,"data_compressed_allocated":6082560,"data_compressed_original":65804640,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38736,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333104,"statfs":{"total":96636764160,"available":96597098496,"internally_reserved":0,"allocated":3465216,"data_stored":23476588,"data_compressed":1169908,"data_compressed_allocated":2035712,"data_compressed_original":22736851,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44320,"kb_used_data":8008,"kb_used_omap":4,"kb_used_meta":36283,"kb_avail":94327520,"statfs":{"total":96636764160,"available":96591380480,"internally_reserved":0,"allocated":8200192,"data_stored":64636746,"data_compressed":2952870,"data_compressed_allocated":5865472,"data_compressed_original":63412478,"omap_allocated":4394,"internal_metadata":37154518},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":5576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":5709824,"data_stored":40706666,"data_compressed":2248429,"data_compressed_allocated":3788800,"data_compressed_original":39640615,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":35643695,"data_compressed":2027336,"data_compressed_allocated":3260416,"data_compressed_original":35162080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4886528,"data_stored":39039354,"data_compressed":1270618,"data_compressed_allocated":3080192,"data_compressed_original":38203347,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7516160,"data_stored":66747123,"data_compressed":3566953,"data_compressed_allocated":6082560,"data_compressed_original":65804640,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7577600,"data_stored":63063883,"data_compressed":3342922,"data_compressed_allocated":5869568,"data_compressed_original":62052864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4796416,"data_stored":40288466,"data_compressed":2248429,"data_compressed_allocated":3788800,"data_compressed_original":39640615,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4726784,"data_stored":35479812,"data_compressed":1121840,"data_compressed_allocated":2863104,"data_compressed_original":34634292,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7172096,"data_stored":63598794,"data_compressed":2946946,"data_compressed_allocated":5828608,"data_compressed_original":62839038,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:55:11.281 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-04T21:55:11.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:55:11.281 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-04T21:55:11.282 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-04T21:55:11.282 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:55:11.299 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:11.299 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:11.313 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:11.355 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:11.353+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:11.377 INFO:tasks.ceph.osd.4:Stopped 2024-09-04T21:55:11.378 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-04T21:55:11.378 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:55:11.378 DEBUG:teuthology.orchestra.run.smithi157:> type ceph-objectstore-tool 2024-09-04T21:55:11.422 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-04T21:55:11.423 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-04T21:55:11.440 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:11.440 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:11.448 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:11.445+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:11.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:11.549 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:11.545+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:11.624 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:11.624 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:11.638 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:11.649 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:11.645+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:11.750 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:11.749+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:11.758 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:11.759 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:11.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:11.861 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:11.861+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:11.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:11.938 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:11.938 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:11.952 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:11.949+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:11.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:12.053 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:12.053+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:12.057 INFO:teuthology.orchestra.run.smithi157.stdout:3.f 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.e 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.9 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:2.5 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.11 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.3 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.19 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.13 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.6 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.a 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:2.6 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.14 2024-09-04T21:55:12.058 INFO:teuthology.orchestra.run.smithi157.stdout:3.1b 2024-09-04T21:55:12.059 INFO:teuthology.orchestra.run.smithi157.stdout:3.1 2024-09-04T21:55:12.059 INFO:teuthology.orchestra.run.smithi157.stdout:3.1a 2024-09-04T21:55:12.059 INFO:teuthology.orchestra.run.smithi157.stdout:3.b 2024-09-04T21:55:12.059 INFO:teuthology.orchestra.run.smithi157.stdout:3.1c 2024-09-04T21:55:12.059 INFO:teuthology.orchestra.run.smithi157.stdout:3.1d 2024-09-04T21:55:12.077 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:12.078 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:12.094 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:12.094 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:12.111 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:12.111 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:12.113 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:12.113 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:12.113 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:12.113 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:12.113 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:12.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:12.128 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:12.149 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:12.154 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:12.149+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:12.170 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:12.171 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:12.191 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:12.191 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:12.195 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:12.245 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:12.245 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.392423285999996, 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:55:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:55:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.266376023000007, 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:55:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:55:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.069822384999995, 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.259 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:55:12.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:55:12.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.042432887000004, 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:55:12.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.269 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.019168415999999, 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:55:12.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:55:12.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.991410626000004, 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:55:12.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:55:12.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:55:12.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.991295063999999, 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.282 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:55:12.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:55:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.691120279000003, 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:55:12.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:55:12.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.531987727000001, 2024-09-04T21:55:12.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:12.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:55:12.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.531761669000005, 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:55:12.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.308 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.054548600999993, 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.309 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:55:12.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:55:12.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.825160490000002, 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.315 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:55:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:55:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.526969848999997, 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:55:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:55:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.526856993999999, 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:55:12.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:55:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.333 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.517196025999993, 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.334 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:55:12.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:55:12.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.497346300999993, 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:12.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:55:12.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:12.343 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.347117615000002, 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:55:12.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.349 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.680790709999997, 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:55:12.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:55:12.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.677651523999998, 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:12.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:55:12.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.659340534000002, 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:55:12.360 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:12.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:55:12.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.365 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:12.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:12.367 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:12.365+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:12.468 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:12.465+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:12.494 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:12.494 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:12.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:12.530 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:55:12.534 INFO:teuthology.orchestra.run.smithi112.stdout:4753 write 0xb584749 thru 0xb584df3 (0x6ab bytes) 2024-09-04T21:55:12.543 INFO:teuthology.orchestra.run.smithi112.stdout:4754 read 0xbcf797 thru 0xbd525e (0x5ac8 bytes) 2024-09-04T21:55:12.544 INFO:teuthology.orchestra.run.smithi112.stdout:4755 punch from 0x5ee008b to 0x5eeaef1, (0xae66 bytes) 2024-09-04T21:55:12.544 INFO:teuthology.orchestra.run.smithi112.stdout:4756 writesame 0x73c876 thru 0x747eda data_size 0x1(0xb665 bytes) 2024-09-04T21:55:12.568 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:12.565+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:12.575 INFO:teuthology.orchestra.run.smithi112.stdout:4757 write 0x89f914 thru 0x8af0c7 (0xf7b4 bytes) 2024-09-04T21:55:12.585 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:12.585 INFO:teuthology.orchestra.run.smithi112.stdout:4759 write 0x1d4eb3d thru 0x1d5d079 (0xe53d bytes) 2024-09-04T21:55:12.593 INFO:teuthology.orchestra.run.smithi112.stdout:4762 write 0x1a9471e thru 0x1a9682d (0x2110 bytes) 2024-09-04T21:55:12.599 INFO:teuthology.orchestra.run.smithi112.stdout:4763 read 0x729fe4b thru 0x72a7ee0 (0x8096 bytes) 2024-09-04T21:55:12.599 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:12.599 INFO:teuthology.orchestra.run.smithi112.stdout:4766 write 0x62181c4 thru 0x62226b6 (0xa4f3 bytes) 2024-09-04T21:55:12.603 INFO:teuthology.orchestra.run.smithi112.stdout:4769 read 0xb4ffdb2 thru 0xb50eecf (0xf11e bytes) 2024-09-04T21:55:12.604 INFO:teuthology.orchestra.run.smithi112.stdout:4770 write 0x4f1b58b thru 0x4f1c09a (0xb10 bytes) 2024-09-04T21:55:12.615 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 3.14 --file /var/log/ceph/exp.3.14.4 2024-09-04T21:55:12.619 INFO:teuthology.orchestra.run.smithi112.stdout:4771 writesame 0xd307caf thru 0xd30c0a1 data_size 0x47(0x43f3 bytes) 2024-09-04T21:55:12.626 INFO:teuthology.orchestra.run.smithi112.stdout:4772 trunc from 0xd30c0a2 to 0xa3a486c 2024-09-04T21:55:12.641 INFO:teuthology.orchestra.run.smithi112.stdout:4773 read 0x6e99d81 thru 0x6e9c440 (0x26c0 bytes) 2024-09-04T21:55:12.641 INFO:teuthology.orchestra.run.smithi112.stdout:4774 clone 28 order 24 su 4194304 sc 14 2024-09-04T21:55:12.643 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:12.643 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:12.643 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:12.643 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:12.643 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:12.658 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:12.701 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:12.701+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:12.770 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:12.769+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:12.817 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32654 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 4800512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226896517999997, 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.027786758000000002, 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32654 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588399+0000", 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588400+0000", 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3399999999999999e-07 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588403+0000", 2024-09-04T21:55:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7249999999999999e-06 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588414+0000", 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0559e-05 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.588421+0000", 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5109999999999997e-06 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590927+0000", 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025057339999999999 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.590933+0000", 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5820000000000001e-06 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610352+0000", 2024-09-04T21:55:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.019419174000000001 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.610442+0000", 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0125999999999997e-05 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.616186+0000", 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057437130000000001 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32670 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5062656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226282722000001, 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024152731, 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32670 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589013+0000", 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4199999999999995e-07 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589017+0000", 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.224e-06 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589031+0000", 2024-09-04T21:55:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4747e-05 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589039+0000", 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591456+0000", 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024168549999999999 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591462+0000", 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4979999999999997e-06 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613066+0000", 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0216037 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613120+0000", 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4209999999999998e-05 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613165+0000", 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5763999999999998e-05 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32671 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5079040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226243314000001, 2024-09-04T21:55:12.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024282133000000001, 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32671 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:12.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589052+0000", 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589053+0000", 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589057+0000", 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589074+0000", 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7201999999999999e-05 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589083+0000", 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5739999999999996e-06 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591487+0000", 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024049150000000001 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591493+0000", 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.592e-06 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613189+0000", 2024-09-04T21:55:12.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021695945000000001 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613260+0000", 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0905e-05 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613334+0000", 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4425999999999995e-05 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.834 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32672 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5095424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226200710999997, 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024400369000000002, 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.835 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32672 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589095+0000", 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1299999999999999e-07 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589099+0000", 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.591e-06 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589115+0000", 2024-09-04T21:55:12.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5591999999999999e-05 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589128+0000", 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3210000000000001e-05 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591520+0000", 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023920529999999999 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591526+0000", 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613358+0000", 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021832949000000001 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613427+0000", 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8955999999999995e-05 2024-09-04T21:55:12.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613495+0000", 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7694999999999995e-05 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32673 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226156580999998, 2024-09-04T21:55:12.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024527538000000002, 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32673 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589139+0000", 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589140+0000", 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589143+0000", 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.331e-06 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589153+0000", 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7270000000000002e-06 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589161+0000", 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8149999999999995e-06 2024-09-04T21:55:12.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591553+0000", 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023916469999999998 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591558+0000", 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1649999999999997e-06 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613520+0000", 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021961729999999999 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613591+0000", 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0735000000000006e-05 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613666+0000", 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5754999999999996e-05 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.845 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32674 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5128192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226123088999998, 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024653952999999999, 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32674 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589172+0000", 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589173+0000", 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7000000000000005e-07 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589186+0000", 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2972999999999999e-05 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589204+0000", 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7666999999999999e-05 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589212+0000", 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3999999999999992e-06 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591589+0000", 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023771370000000001 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591594+0000", 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3149999999999999e-06 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613690+0000", 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022095789000000001 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613759+0000", 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9142999999999999e-05 2024-09-04T21:55:12.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613826+0000", 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6958999999999994e-05 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32675 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5144576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226070669999999, 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024775136, 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.851 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32675 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589225+0000", 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589226+0000", 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0500000000000003e-07 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589229+0000", 2024-09-04T21:55:12.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999998e-06 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589239+0000", 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589247+0000", 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0430000000000008e-06 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591622+0000", 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023745099999999998 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591627+0000", 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5300000000000004e-06 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613851+0000", 2024-09-04T21:55:12.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022223613 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.613923+0000", 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2203e-05 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614000+0000", 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7095000000000006e-05 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32676 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5160960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.226037433000002, 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024914054000000001, 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32676 2024-09-04T21:55:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589258+0000", 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589259+0000", 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589276+0000", 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7686e-05 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589288+0000", 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1234e-05 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589296+0000", 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5769999999999994e-06 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591654+0000", 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023581769999999999 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591660+0000", 2024-09-04T21:55:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3109999999999996e-06 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614025+0000", 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022365051 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614099+0000", 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3881000000000006e-05 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614172+0000", 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3498999999999995e-05 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32677 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5177344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.225979402, 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025050084, 2024-09-04T21:55:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32677 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589316+0000", 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589317+0000", 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5300000000000003e-07 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589329+0000", 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1951e-05 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589346+0000", 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7108e-05 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589354+0000", 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2630000000000003e-06 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591674+0000", 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023197980000000001 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591677+0000", 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5909999999999998e-06 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614205+0000", 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022528256999999999 2024-09-04T21:55:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614287+0000", 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1886999999999997e-05 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614366+0000", 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9475999999999996e-05 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32678 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5193728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.225927210000002, 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025172158, 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32678 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589368+0000", 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589369+0000", 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8400000000000004e-07 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589373+0000", 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589391+0000", 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8077e-05 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589400+0000", 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0790000000000007e-06 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591689+0000", 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002288815 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591692+0000", 2024-09-04T21:55:12.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.666e-06 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614392+0000", 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022700287 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614462+0000", 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0468000000000004e-05 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614540+0000", 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8104999999999998e-05 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32679 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5210112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.225883277000001, 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025302998, 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32679 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589412+0000", 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589413+0000", 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0100000000000004e-07 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589417+0000", 2024-09-04T21:55:12.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7689999999999998e-06 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589435+0000", 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8185e-05 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589444+0000", 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4029999999999996e-06 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591704+0000", 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022601729999999999 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591708+0000", 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.067e-06 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614571+0000", 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022863270000000002 2024-09-04T21:55:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614640+0000", 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8909e-05 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614715+0000", 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5520999999999997e-05 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32680 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5226496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.225837667, 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.025450210000000001, 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32680 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.879 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589458+0000", 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589459+0000", 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589462+0000", 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7780000000000001e-06 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589479+0000", 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6651e-05 2024-09-04T21:55:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.589488+0000", 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6309999999999992e-06 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591737+0000", 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002249301 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.591742+0000", 2024-09-04T21:55:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4790000000000004e-06 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614747+0000", 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.023004539000000001 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614825+0000", 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8566000000000006e-05 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.614908+0000", 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2558999999999993e-05 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32930 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5136384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216200078999996, 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024238576000000001, 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32930 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599095+0000", 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599096+0000", 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5699999999999998e-07 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599098+0000", 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2979999999999999e-06 2024-09-04T21:55:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599105+0000", 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5479999999999998e-06 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599110+0000", 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1109999999999999e-06 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601491+0000", 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023813800000000002 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601496+0000", 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623247+0000", 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021751177 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623291+0000", 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4184999999999998e-05 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623334+0000", 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2970999999999999e-05 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32931 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5152768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216177813000002, 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024330243000000001, 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32931 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599118+0000", 2024-09-04T21:55:12.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.61e-07 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599121+0000", 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.683e-06 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599129+0000", 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6359999999999994e-06 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599134+0000", 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5049999999999998e-06 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601519+0000", 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002384546 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601521+0000", 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2919999999999998e-06 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623360+0000", 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021839164000000001 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623405+0000", 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4243999999999999e-05 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623448+0000", 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3411999999999997e-05 2024-09-04T21:55:12.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32932 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5169152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216152778999998, 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024418507999999998, 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.895 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32932 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599143+0000", 2024-09-04T21:55:12.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599144+0000", 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4799999999999997e-07 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599153+0000", 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9930000000000001e-06 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599161+0000", 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0900000000000005e-06 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599167+0000", 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0929999999999999e-06 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601543+0000", 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023758540000000002 2024-09-04T21:55:12.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601547+0000", 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1019999999999999e-06 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623474+0000", 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.021927782 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623519+0000", 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4263e-05 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623561+0000", 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2483e-05 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32933 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5185536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216118684000001, 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024496766999999999, 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32933 2024-09-04T21:55:12.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599177+0000", 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4700000000000001e-07 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599180+0000", 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8839999999999998e-06 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599188+0000", 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6450000000000005e-06 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599203+0000", 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5099999999999999e-05 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601573+0000", 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002369566 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601577+0000", 2024-09-04T21:55:12.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5090000000000004e-06 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623587+0000", 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022009635999999999 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623631+0000", 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4605999999999999e-05 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623674+0000", 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2274000000000002e-05 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32934 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5201920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216082512, 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024572765999999999, 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32934 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599213+0000", 2024-09-04T21:55:12.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599214+0000", 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599217+0000", 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.095e-06 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599224+0000", 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6760000000000007e-06 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599230+0000", 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8039999999999998e-06 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601603+0000", 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023727090000000002 2024-09-04T21:55:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601608+0000", 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8500000000000002e-06 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623700+0000", 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022092219999999999 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623744+0000", 2024-09-04T21:55:12.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4013e-05 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623786+0000", 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1749000000000003e-05 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32935 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5218304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216056227999999, 2024-09-04T21:55:12.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024658065999999999, 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32935 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:12.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599239+0000", 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599240+0000", 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3500000000000002e-07 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599249+0000", 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4620000000000002e-06 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599258+0000", 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7069999999999998e-06 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599264+0000", 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4790000000000001e-06 2024-09-04T21:55:12.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601634+0000", 2024-09-04T21:55:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023694020000000001 2024-09-04T21:55:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601638+0000", 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623812+0000", 2024-09-04T21:55:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022173986 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623856+0000", 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3696000000000003e-05 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623897+0000", 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1530999999999999e-05 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32936 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:f [read 5234688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.216020876999998, 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.024740649, 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32936 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599275+0000", 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2099999999999997e-07 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599278+0000", 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8870000000000001e-06 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599285+0000", 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4449999999999999e-06 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.599291+0000", 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7649999999999999e-06 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601668+0000", 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023769379999999999 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.601673+0000", 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8210000000000001e-06 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623927+0000", 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.022253536000000001 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.623972+0000", 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5216999999999998e-05 2024-09-04T21:55:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.624015+0000", 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3519000000000001e-05 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:106971 3.0 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head [stat,delete] snapc 1c=[1c] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:08.629998+0000", 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.1852974789999999, 2024-09-04T21:55:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046612358, 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 106971 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.629998+0000", 2024-09-04T21:55:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.629998+0000", 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630001+0000", 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.976e-06 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630010+0000", 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9600000000000006e-06 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630031+0000", 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0647000000000001e-05 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630040+0000", 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3379999999999994e-06 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630121+0000", 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0988000000000005e-05 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630216+0000", 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5320000000000002e-05 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6", 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.630372+0000", 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156119 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.646312+0000", 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015939512999999999 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.676524+0000", 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.030212063000000001 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.676570+0000", 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6119999999999999e-05 2024-09-04T21:55:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:08.676610+0000", 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0314000000000001e-05 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:12.929 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:12.931 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:12.931 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:12.929+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:13.031 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:13.029+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:13.098 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:13.098 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:13.112 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:13.153 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:13.153+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:13.208 INFO:teuthology.orchestra.run.smithi157.stderr:Exporting 3.14 info 3.14( v 84'164 (0'0,84'164] local-lis/les=48/49 n=31 ec=22/22 lis/c=48/48 les/c/f=49/49/0 sis=48) 2024-09-04T21:55:13.210 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:284215d9:::rbd_data.1188e7c968d9.0000000000000002:4# 2024-09-04T21:55:13.211 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:284215d9:::rbd_data.1188e7c968d9.0000000000000002:head# 2024-09-04T21:55:13.211 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2850fa02:::rbd_data.118816ab2dec.000000000000016f:1b# 2024-09-04T21:55:13.211 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2850fa02:::rbd_data.118816ab2dec.000000000000016f:head# 2024-09-04T21:55:13.211 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2891a0a1:::rbd_data.11882296a3bb.0000000000000048:12# 2024-09-04T21:55:13.212 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2891a0a1:::rbd_data.11882296a3bb.0000000000000048:head# 2024-09-04T21:55:13.212 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:297090b0:::rbd_data.1188b450424.0000000000000017:14# 2024-09-04T21:55:13.213 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:297090b0:::rbd_data.1188b450424.0000000000000017:head# 2024-09-04T21:55:13.213 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:29898bb4:::rbd_data.1188b450424.0000000000000018:14# 2024-09-04T21:55:13.213 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:29898bb4:::rbd_data.1188b450424.0000000000000018:head# 2024-09-04T21:55:13.214 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2a299d23:::rbd_data.1188e7c968d9.000000000000007c:4# 2024-09-04T21:55:13.215 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2a299d23:::rbd_data.1188e7c968d9.000000000000007c:head# 2024-09-04T21:55:13.215 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2acf6680:::rbd_object_map.11883c2bd208:head# 2024-09-04T21:55:13.215 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2af03d74:::rbd_data.1188388ff6d.000000000000006b:head# 2024-09-04T21:55:13.215 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2afd88fa:::rbd_data.1188c8e6983d.0000000000000000:head# 2024-09-04T21:55:13.215 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2b5a5038:::rbd_data.1188e5a3a295.0000000000000001:18# 2024-09-04T21:55:13.218 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2b5a5038:::rbd_data.1188e5a3a295.0000000000000001:head# 2024-09-04T21:55:13.218 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2ba9bd77:::rbd_data.118816ab2dec.0000000000000048:1b# 2024-09-04T21:55:13.219 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2ba9bd77:::rbd_data.118816ab2dec.0000000000000048:head# 2024-09-04T21:55:13.219 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2bb4acb3:::rbd_id.image_client.0-clone5:head# 2024-09-04T21:55:13.219 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2bb82ebc:::rbd_data.11883c2bd208.0000000000000045:head# 2024-09-04T21:55:13.219 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2bc09623:::rbd_data.1188b450424.0000000000000055:14# 2024-09-04T21:55:13.220 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2bc09623:::rbd_data.1188b450424.0000000000000055:head# 2024-09-04T21:55:13.220 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2bc25c91:::rbd_data.1188aa2c625d.0000000000000023:head# 2024-09-04T21:55:13.222 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2cfeb677:::rbd_data.118816ab2dec.0000000000000171:1b# 2024-09-04T21:55:13.222 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2cfeb677:::rbd_data.118816ab2dec.0000000000000171:head# 2024-09-04T21:55:13.222 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2d87aeef:::rbd_data.1188388ff6d.0000000000000005:head# 2024-09-04T21:55:13.222 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2de59dee:::rbd_data.11885236e45a.00000000000000e7:head# 2024-09-04T21:55:13.222 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2e0da751:::rbd_object_map.1188c8e6983d.0000000000000017:head# 2024-09-04T21:55:13.222 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2faad357:::rbd_data.1188e5a3a295.0000000000000002:18# 2024-09-04T21:55:13.223 INFO:teuthology.orchestra.run.smithi157.stderr:Read #3:2faad357:::rbd_data.1188e5a3a295.0000000000000002:head# 2024-09-04T21:55:13.223 INFO:teuthology.orchestra.run.smithi157.stderr:Export successful 2024-09-04T21:55:13.233 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:13.233+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:13.274 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:13.274 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:13.274 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:13.274 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:13.274 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:13.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:13.373 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:13.373+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:13.435 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:13.432+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.954891244000002, 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.610114857999999, 2024-09-04T21:55:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:13.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:13.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.610086252999999, 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.610066216, 2024-09-04T21:55:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.610046994000001, 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:13.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:13.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:13.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.498 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.610028202999999, 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.499 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:13.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609996260999999, 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.508 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609967732000001, 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:13.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:13.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:13.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:13.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:13.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:13.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.514 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609948120999999, 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.515 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:13.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:13.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609929827999999, 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:13.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:13.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.525 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:13.526 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609907014000001, 2024-09-04T21:55:13.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:13.526 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.526 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:13.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:13.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:13.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.532 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609887876999998, 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:13.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:13.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.537 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609867211000001, 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.542 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609836434000002, 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.546 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609817066000002, 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:13.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:13.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609793107000002, 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.552 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:13.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609745858, 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:13.556 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:13.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.60972018, 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609686153999998, 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.609661753000001, 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:13.574 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:13.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 3.14 2024-09-04T21:55:13.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:13.577 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:55:13.577 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:13.572+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:13.676 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:13.672+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:13.747 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone27 from 0xa3a486c (overlap 0x23c0c48) to 0x1c9a36c 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:123310 3.1f e87/67 3:fbbea59d:::rbd_data.1188e6911c8f.0000000000000044:head v 87'187, mlcod=84'185)", 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:13.761030+0000", 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.00077684399999999995, 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082453500000000005, 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:13.762 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761030+0000", 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761030+0000", 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:13.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761031+0000", 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0279999999999999e-06 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761041+0000", 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0343000000000001e-05 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761055+0000", 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3906e-05 2024-09-04T21:55:13.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761062+0000", 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8379999999999996e-06 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761169+0000", 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106606 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761221+0000", 2024-09-04T21:55:13.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1841000000000001e-05 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:123313 3.d e87/48 3:b03af129:::rbd_data.1188e6911c8f.000000000000003f:head v 87'268, mlcod=87'266)", 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:13.761111+0000", 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.00069585800000000005, 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084253099999999999, 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:13.766 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761111+0000", 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761111+0000", 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761112+0000", 2024-09-04T21:55:13.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0699999999999996e-07 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761117+0000", 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3340000000000001e-06 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761128+0000", 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0535e-05 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:13.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761135+0000", 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8730000000000007e-06 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761264+0000", 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012847800000000001 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761290+0000", 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:13.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:13.761955+0000", 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066500999999999995 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 2 2024-09-04T21:55:13.770 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:13.777 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:13.776+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:13.779 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:13.878 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:13.872+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:13.895 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:13.895 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:13.895 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:13.895 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:13.895 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:13.910 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:13.995 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:13.992+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:14.071 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-09-04T21:55:14.074 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 48.631847327000003, 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:14.075 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:55:14.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:55:14.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 48.447306271999999, 2024-09-04T21:55:14.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:14.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:55:14.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:55:14.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:55:14.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:14.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43351 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.114010843000003, 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051338846, 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43351 2024-09-04T21:55:14.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4499999999999997e-07 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959605+0000", 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959613+0000", 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8250000000000005e-06 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959618+0000", 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.417e-06 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010746+0000", 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051127924999999998 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010835+0000", 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9444000000000006e-05 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010941+0000", 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105312 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43353 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1269760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:14.092 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113986791000002, 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051544344999999998, 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43353 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:14.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959629+0000", 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959635+0000", 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0430000000000004e-06 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959640+0000", 2024-09-04T21:55:14.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0320000000000003e-06 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010976+0000", 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051336248000000001 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011067+0000", 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0826000000000001e-05 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011170+0000", 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103507 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43355 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113964649000003, 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051760575000000003, 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.097 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43355 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7800000000000002e-07 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959651+0000", 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959658+0000", 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5730000000000002e-06 2024-09-04T21:55:14.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959663+0000", 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011205+0000", 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051541489000000003 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011296+0000", 2024-09-04T21:55:14.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1105000000000003e-05 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011409+0000", 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112927 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43358 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113941683999997, 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051974999000000001, 2024-09-04T21:55:14.102 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43358 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9999999999999998e-07 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959674+0000", 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959682+0000", 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9500000000000001e-06 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959687+0000", 2024-09-04T21:55:14.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1420000000000001e-06 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011451+0000", 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051763862000000001 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011538+0000", 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7322000000000002e-05 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011646+0000", 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010772000000000001 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43361 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113918267000003, 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052194391, 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43361 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959695+0000", 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959697+0000", 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.418e-06 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959703+0000", 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9989999999999997e-06 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959708+0000", 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.904e-06 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011687+0000", 2024-09-04T21:55:14.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051978506000000001 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011781+0000", 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4301000000000004e-05 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011889+0000", 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107796 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43362 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113897475999998, 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052922181999999998, 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.112 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43362 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959716+0000", 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959718+0000", 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959725+0000", 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4660000000000002e-06 2024-09-04T21:55:14.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959730+0000", 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1019999999999996e-06 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011924+0000", 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052193145000000003 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012015+0000", 2024-09-04T21:55:14.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0969000000000003e-05 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012637+0000", 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062277700000000003 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43364 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1351680~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113875073000003, 2024-09-04T21:55:14.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052915188000000002, 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43364 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:14.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3799999999999997e-07 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959740+0000", 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959747+0000", 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9149999999999999e-06 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959754+0000", 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8000000000000001e-06 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012425+0000", 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052670531 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012525+0000", 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100581 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012653+0000", 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012751700000000001 2024-09-04T21:55:14.120 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43366 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113851425, 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052927226000000001, 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.121 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43366 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959762+0000", 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0399999999999996e-07 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959764+0000", 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3949999999999999e-06 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960370+0000", 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060537999999999998 2024-09-04T21:55:14.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960389+0000", 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9326000000000001e-05 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012680+0000", 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052291232 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012688+0000", 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3890000000000005e-06 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43368 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.113192740000002, 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052287419000000002, 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43368 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:14.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960422+0000", 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.84e-06 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960431+0000", 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960971+0000", 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000539965 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960991+0000", 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0832999999999999e-05 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012703+0000", 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051711615000000002 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012707+0000", 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3499999999999999e-06 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43369 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112592034000002, 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051701824, 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.129 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43369 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961022+0000", 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6139999999999999e-06 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961030+0000", 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1489999999999994e-06 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961051+0000", 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961065+0000", 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4163e-05 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012718+0000", 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051653533000000001 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012722+0000", 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43371 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112526971999998, 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051652328999999997, 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.133 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43371 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961087+0000", 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961106+0000", 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9097e-05 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961146+0000", 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0281999999999998e-05 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961162+0000", 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5993000000000001e-05 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012734+0000", 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051571744000000003 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012738+0000", 2024-09-04T21:55:14.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:14.137 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.139 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43373 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112429265000003, 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051570802999999998, 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43373 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961185+0000", 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.423e-06 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961191+0000", 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9429999999999997e-06 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961210+0000", 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8943000000000001e-05 2024-09-04T21:55:14.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961223+0000", 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3506000000000001e-05 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012750+0000", 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051527008999999999 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012754+0000", 2024-09-04T21:55:14.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9790000000000004e-06 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43375 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112370089999999, 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051526660000000002, 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.144 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43375 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961244+0000", 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961249+0000", 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961270+0000", 2024-09-04T21:55:14.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0925999999999999e-05 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961283+0000", 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2986000000000001e-05 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012765+0000", 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0514825 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012769+0000", 2024-09-04T21:55:14.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8079999999999998e-06 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43377 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112312854999999, 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051486632999999997, 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.148 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43377 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:14.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961301+0000", 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961306+0000", 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.69e-06 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961323+0000", 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6976000000000001e-05 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961336+0000", 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2913e-05 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012781+0000", 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051444996999999999 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012786+0000", 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9340000000000003e-06 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43379 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112260208999999, 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051454868000000001, 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.152 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43379 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961354+0000", 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0729999999999999e-06 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961358+0000", 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6099999999999999e-06 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961375+0000", 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7091999999999998e-05 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961390+0000", 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4803e-05 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012803+0000", 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051413343 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012807+0000", 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9469999999999996e-06 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.156 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43381 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112203780999998, 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051415832000000002, 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43381 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961410+0000", 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961415+0000", 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.977e-06 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961431+0000", 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6016e-05 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961444+0000", 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2816000000000001e-05 2024-09-04T21:55:14.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012821+0000", 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051376809000000002 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012825+0000", 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0910000000000003e-06 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43383 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112150653, 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051378030999999998, 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43383 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961463+0000", 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961473+0000", 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961535+0000", 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2787999999999996e-05 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961555+0000", 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9646000000000001e-05 2024-09-04T21:55:14.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012836+0000", 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051281032999999997 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012840+0000", 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9840000000000003e-06 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43385 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.112022705999998, 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051265590999999999, 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43385 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:14.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961591+0000", 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-04T21:55:14.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961597+0000", 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961615+0000", 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7836e-05 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961628+0000", 2024-09-04T21:55:14.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2976000000000001e-05 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012851+0000", 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051223109000000003 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012856+0000", 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7870000000000001e-06 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.168 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:14.169 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:14.169 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:14.170 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:14.170 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:14.164+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:14.196 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:14.270 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:14.268+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:14.341 INFO:teuthology.orchestra.run.smithi112.stdout:4775 punch from 0x52fc5f1 to 0x5303620, (0x702f bytes) 2024-09-04T21:55:14.348 INFO:teuthology.orchestra.run.smithi112.stdout:4777 writesame 0x4a21fc0 thru 0x4a25cd9 data_size 0x16(0x3d1a bytes) 2024-09-04T21:55:14.349 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:14.349 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:14.349 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:14.349 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:14.357 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:14.366 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:14.371 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:14.368+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:14.374 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:14.388 INFO:teuthology.orchestra.run.smithi112.stdout:4779 trunc from 0xa3a486c to 0x48d0787 2024-09-04T21:55:14.394 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:14.394 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:14.419 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:14.419 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:14.419 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:14.443 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:14.443 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:14.456 INFO:teuthology.orchestra.run.smithi112.stdout:4782 writesame 0x141e527 thru 0x1420ef5 data_size 0xb(0x29cf bytes) 2024-09-04T21:55:14.458 INFO:teuthology.orchestra.run.smithi112.stdout:4783 read 0x3ad99e8 thru 0x3ae6b6d (0xd186 bytes) 2024-09-04T21:55:14.461 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:14.461 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:14.471 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:14.468+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:14.473 INFO:teuthology.orchestra.run.smithi112.stdout:4785 write 0xb38d784 thru 0xb38df42 (0x7bf bytes) 2024-09-04T21:55:14.475 INFO:teuthology.orchestra.run.smithi112.stdout:4787 write 0x8cebd1 thru 0x8dc2ab (0xd6db bytes) 2024-09-04T21:55:14.476 INFO:teuthology.orchestra.run.smithi112.stdout:4788 writesame 0x4db8ab2 thru 0x4dc2743 data_size 0x31(0x9c92 bytes) 2024-09-04T21:55:14.479 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:14.479 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:14.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:14.486 INFO:teuthology.orchestra.run.smithi112.stdout:4790 write 0xc58985f thru 0xc5907d7 (0x6f79 bytes) 2024-09-04T21:55:14.488 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.488 INFO:teuthology.orchestra.run.smithi112.stdout:4792 write 0x7d2bc4 thru 0x7df1e1 (0xc61e bytes) 2024-09-04T21:55:14.490 INFO:teuthology.orchestra.run.smithi112.stdout:4793 write 0x70c7a8 thru 0x718473 (0xbccc bytes) 2024-09-04T21:55:14.492 INFO:teuthology.orchestra.run.smithi112.stdout:4794 write 0x5509e00 thru 0x550d8a1 (0x3aa2 bytes) 2024-09-04T21:55:14.494 INFO:teuthology.orchestra.run.smithi112.stdout:4795 write 0x312a453 thru 0x312b138 (0xce6 bytes) 2024-09-04T21:55:14.495 INFO:teuthology.orchestra.run.smithi112.stdout:4796 read 0x4ad35a9 thru 0x4ad5689 (0x20e1 bytes) 2024-09-04T21:55:14.495 INFO:teuthology.orchestra.run.smithi112.stdout:4797 read 0x1b511b1 thru 0x1b5f7cd (0xe61d bytes) 2024-09-04T21:55:14.504 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:14.504 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:14.507 INFO:teuthology.orchestra.run.smithi112.stdout:4799 trunc from 0xc5907d8 to 0x571fb79 2024-09-04T21:55:14.516 INFO:teuthology.orchestra.run.smithi112.stdout:4801 write 0x4827d61 thru 0x4831265 (0x9505 bytes) 2024-09-04T21:55:14.522 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:14.537 INFO:teuthology.orchestra.run.smithi112.stdout:4803 write 0xcbf87b5 thru 0xcbfaeee (0x273a bytes) 2024-09-04T21:55:14.539 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.539 INFO:teuthology.orchestra.run.smithi112.stdout:4805 writesame 0x460909f thru 0x4616ae6 data_size 0x7f(0xda48 bytes) 2024-09-04T21:55:14.541 INFO:teuthology.orchestra.run.smithi112.stdout:4807 write 0x97cfa9c thru 0x97d8060 (0x85c5 bytes) 2024-09-04T21:55:14.542 INFO:teuthology.orchestra.run.smithi112.stdout:4809 writesame 0x1335a4f thru 0x1336006 data_size 0xc(0x5b8 bytes) 2024-09-04T21:55:14.544 INFO:teuthology.orchestra.run.smithi112.stdout:4810 read 0x79fd96 thru 0x7af8fa (0xfb65 bytes) 2024-09-04T21:55:14.545 INFO:teuthology.orchestra.run.smithi112.stdout:4812 punch from 0xbaf431 to 0xbb6c32, (0x7801 bytes) 2024-09-04T21:55:14.546 INFO:teuthology.orchestra.run.smithi112.stdout:4814 write 0x4390871 thru 0x439e2b8 (0xda48 bytes) 2024-09-04T21:55:14.557 INFO:teuthology.orchestra.run.smithi112.stdout:4815 write 0xe057914 thru 0xe05b461 (0x3b4e bytes) 2024-09-04T21:55:14.562 INFO:teuthology.orchestra.run.smithi112.stdout:4816 trunc from 0xe05b462 to 0xe2e915e 2024-09-04T21:55:14.571 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.571 INFO:teuthology.orchestra.run.smithi112.stdout:4819 write 0x8161c60 thru 0x816d7de (0xbb7f bytes) 2024-09-04T21:55:14.572 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:14.568+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:14.586 INFO:teuthology.orchestra.run.smithi112.stdout:4820 trunc from 0xe2e915e to 0x64aa81d 2024-09-04T21:55:14.626 INFO:teuthology.orchestra.run.smithi112.stdout:4821 write 0xcdec43d thru 0xcdf2cce (0x6892 bytes) 2024-09-04T21:55:14.628 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.645 INFO:teuthology.orchestra.run.smithi112.stdout:4824 trunc from 0xcdf2ccf to 0x46743e1 2024-09-04T21:55:14.648 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:14.648 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:14.659 INFO:teuthology.orchestra.run.smithi112.stdout:4827 read 0x3adc2de thru 0x3adf7e9 (0x350c bytes) 2024-09-04T21:55:14.660 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.665 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:14.673 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:14.672+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:14.680 INFO:teuthology.orchestra.run.smithi112.stdout:4829 write 0x9c5a54d thru 0x9c629d0 (0x8484 bytes) 2024-09-04T21:55:14.707 INFO:teuthology.orchestra.run.smithi112.stdout:4831 writesame 0xee48b58 thru 0xee5321c data_size 0x39(0xa6c5 bytes) 2024-09-04T21:55:14.707 INFO:teuthology.orchestra.run.smithi112.stdout:4834 read 0x2a5b38 thru 0x2a60fd (0x5c6 bytes) 2024-09-04T21:55:14.708 INFO:teuthology.orchestra.run.smithi112.stdout:4835 punch from 0x5802fc4 to 0x580f142, (0xc17e bytes) 2024-09-04T21:55:14.708 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.708 INFO:teuthology.orchestra.run.smithi112.stdout:4837 read 0x1b195d6 thru 0x1b20636 (0x7061 bytes) 2024-09-04T21:55:14.710 INFO:teuthology.orchestra.run.smithi112.stdout:4839 punch from 0x74c09b4 to 0x74cbc9d, (0xb2e9 bytes) 2024-09-04T21:55:14.710 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.728 INFO:teuthology.orchestra.run.smithi112.stdout:4843 trunc from 0xee5321d to 0x701b92c 2024-09-04T21:55:14.741 INFO:teuthology.orchestra.run.smithi112.stdout:4844 read 0x4d3ab38 thru 0x4d49158 (0xe621 bytes) 2024-09-04T21:55:14.741 INFO:teuthology.orchestra.run.smithi112.stdout:4845 read 0x5682a91 thru 0x5685151 (0x26c1 bytes) 2024-09-04T21:55:14.742 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.743 INFO:teuthology.orchestra.run.smithi112.stdout:4847 write 0x1715eaf thru 0x1723eef (0xe041 bytes) 2024-09-04T21:55:14.766 INFO:teuthology.orchestra.run.smithi112.stdout:4848 write 0x9dfed10 thru 0x9dff6ba (0x9ab bytes) 2024-09-04T21:55:14.770 INFO:teuthology.orchestra.run.smithi112.stdout:4849 punch from 0x10a1c9c to 0x10a9375, (0x76d9 bytes) 2024-09-04T21:55:14.770 INFO:teuthology.orchestra.run.smithi112.stdout:4850 read 0x134d1f4 thru 0x135077b (0x3588 bytes) 2024-09-04T21:55:14.771 INFO:teuthology.orchestra.run.smithi112.stdout:4851 write 0x6cf28d6 thru 0x6d0061f (0xdd4a bytes) 2024-09-04T21:55:14.773 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:14.768+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:14.774 INFO:teuthology.orchestra.run.smithi112.stdout:4852 punch from 0x68408f to 0x68d274, (0x91e5 bytes) 2024-09-04T21:55:14.774 INFO:teuthology.orchestra.run.smithi112.stdout:4853 writesame 0x390ba47 thru 0x39129bc data_size 0xb(0x6f76 bytes) 2024-09-04T21:55:14.778 INFO:teuthology.orchestra.run.smithi112.stdout:4855 punch from 0x1d22e6e to 0x1d2bf01, (0x9093 bytes) 2024-09-04T21:55:14.778 INFO:teuthology.orchestra.run.smithi112.stdout:4856 read 0x6d56768 thru 0x6d5f8a9 (0x9142 bytes) 2024-09-04T21:55:14.779 INFO:teuthology.orchestra.run.smithi112.stdout:4857 read 0x3d4fdc6 thru 0x3d57003 (0x723e bytes) 2024-09-04T21:55:14.790 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:14.790 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:14.802 INFO:teuthology.orchestra.run.smithi112.stdout:4859 writesame 0xd64693d thru 0xd647d21 data_size 0x1(0x13e5 bytes) 2024-09-04T21:55:14.805 INFO:teuthology.orchestra.run.smithi112.stdout:4860 read 0xb731dbd thru 0xb732c69 (0xead bytes) 2024-09-04T21:55:14.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:14.806 INFO:teuthology.orchestra.run.smithi112.stdout:4861 writesame 0xa6b6309 thru 0xa6ba5a9 data_size 0x1(0x42a1 bytes) 2024-09-04T21:55:14.830 INFO:teuthology.orchestra.run.smithi112.stdout:4862 write 0xe7e983d thru 0xe7f741d (0xdbe1 bytes) 2024-09-04T21:55:14.836 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.836 INFO:teuthology.orchestra.run.smithi112.stdout:4864 write 0x82cec50 thru 0x82d8e80 (0xa231 bytes) 2024-09-04T21:55:14.837 INFO:teuthology.orchestra.run.smithi112.stdout:4865 punch from 0x269a38a to 0x26a362a, (0x92a0 bytes) 2024-09-04T21:55:14.838 INFO:teuthology.orchestra.run.smithi112.stdout:4866 punch from 0x9366f00 to 0x9375d01, (0xee01 bytes) 2024-09-04T21:55:14.838 INFO:teuthology.orchestra.run.smithi112.stdout:4868 write 0xc258dd4 thru 0xc25b612 (0x283f bytes) 2024-09-04T21:55:14.840 INFO:teuthology.orchestra.run.smithi112.stdout:4869 punch from 0x17713f9 to 0x177d2c7, (0xbece bytes) 2024-09-04T21:55:14.840 INFO:teuthology.orchestra.run.smithi112.stdout:4870 read 0x19c5750 thru 0x19d01d1 (0xaa82 bytes) 2024-09-04T21:55:14.841 INFO:teuthology.orchestra.run.smithi112.stdout:4871 read 0x2840b68 thru 0x2841a5f (0xef8 bytes) 2024-09-04T21:55:14.842 INFO:teuthology.orchestra.run.smithi112.stdout:4872 write 0x5f22d58 thru 0x5f293e7 (0x6690 bytes) 2024-09-04T21:55:14.844 INFO:teuthology.orchestra.run.smithi112.stdout:4873 read 0x9bb58ef thru 0x9bc012b (0xa83d bytes) 2024-09-04T21:55:14.845 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.846 INFO:teuthology.orchestra.run.smithi112.stdout:4877 punch from 0xa0a289a to 0xa0ae7aa, (0xbf10 bytes) 2024-09-04T21:55:14.846 INFO:teuthology.orchestra.run.smithi112.stdout:4878 write 0x353e46a thru 0x3549e24 (0xb9bb bytes) 2024-09-04T21:55:14.848 INFO:teuthology.orchestra.run.smithi112.stdout:4880 read 0xc76bdb1 thru 0xc772f19 (0x7169 bytes) 2024-09-04T21:55:14.849 INFO:teuthology.orchestra.run.smithi112.stdout:4881 read 0x1462c4b thru 0x147035a (0xd710 bytes) 2024-09-04T21:55:14.851 INFO:teuthology.orchestra.run.smithi112.stdout:4883 write 0x4a31171 thru 0x4a3b730 (0xa5c0 bytes) 2024-09-04T21:55:14.854 INFO:teuthology.orchestra.run.smithi112.stdout:4884 write 0xd6d5825 thru 0xd6dbef7 (0x66d3 bytes) 2024-09-04T21:55:14.858 INFO:teuthology.orchestra.run.smithi112.stdout:4885 read 0x6ef796d thru 0x6efa4af (0x2b43 bytes) 2024-09-04T21:55:14.874 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:14.872+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:14.876 INFO:teuthology.orchestra.run.smithi112.stdout:4886 trunc from 0xe7f741e to 0x7faefbb 2024-09-04T21:55:14.909 INFO:teuthology.orchestra.run.smithi112.stdout:4890 trunc from 0x7faefbb to 0xd6f9b47 2024-09-04T21:55:14.918 INFO:teuthology.orchestra.run.smithi112.stdout:4893 write 0x8a60666 thru 0x8a60b95 (0x530 bytes) 2024-09-04T21:55:14.923 INFO:teuthology.orchestra.run.smithi112.stdout:4895 trunc from 0xd6f9b47 to 0xc4a06c3 2024-09-04T21:55:14.937 INFO:teuthology.orchestra.run.smithi112.stdout:4896 punch from 0x8e5d7a0 to 0x8e5ec26, (0x1486 bytes) 2024-09-04T21:55:14.937 INFO:teuthology.orchestra.run.smithi112.stdout:4897 writesame 0x7cdcb6f thru 0x7ceb7e7 data_size 0x11(0xec79 bytes) 2024-09-04T21:55:14.941 INFO:teuthology.orchestra.run.smithi112.stdout:4898 write 0x632385f thru 0x632a325 (0x6ac7 bytes) 2024-09-04T21:55:14.945 INFO:teuthology.orchestra.run.smithi112.stdout:4899 punch from 0xb42890c to 0xb42d0c2, (0x47b6 bytes) 2024-09-04T21:55:14.945 INFO:teuthology.orchestra.run.smithi112.stdout:4900 read 0xac132c5 thru 0xac14ae3 (0x181f bytes) 2024-09-04T21:55:14.945 INFO:teuthology.orchestra.run.smithi112.stdout:4903 read 0x12e2b19 thru 0x12ec465 (0x994d bytes) 2024-09-04T21:55:14.947 INFO:teuthology.orchestra.run.smithi112.stdout:4904 read 0x7d97425 thru 0x7da1ae1 (0xa6bd bytes) 2024-09-04T21:55:14.948 INFO:teuthology.orchestra.run.smithi112.stdout:4906 read 0x37db078 thru 0x37e6ce3 (0xbc6c bytes) 2024-09-04T21:55:14.949 INFO:teuthology.orchestra.run.smithi112.stdout:4907 read 0x21d0eb6 thru 0x21d7c68 (0x6db3 bytes) 2024-09-04T21:55:14.962 INFO:teuthology.orchestra.run.smithi112.stdout:4908 trunc from 0xc4a06c3 to 0x814276b 2024-09-04T21:55:14.974 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.975 INFO:teuthology.orchestra.run.smithi112.stdout:4910 read 0x3c8c3db thru 0x3c8c455 (0x7b bytes) 2024-09-04T21:55:14.975 INFO:teuthology.orchestra.run.smithi112.stdout:4911 read 0x33af8ad thru 0x33afbaa (0x2fe bytes) 2024-09-04T21:55:14.975 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:14.972+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:14.976 INFO:teuthology.orchestra.run.smithi112.stdout:4912 writesame 0x36fdd80 thru 0x36ffded data_size 0x7(0x206e bytes) 2024-09-04T21:55:14.978 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:14.989 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:14.989 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:14.992 INFO:teuthology.orchestra.run.smithi112.stdout:4915 trunc from 0x814276b to 0x2211b34 2024-09-04T21:55:15.010 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:15.014 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:15.014 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:15.014 INFO:teuthology.orchestra.run.smithi112.stdout:4919 punch from 0x10fdd1d to 0x11044f0, (0x67d3 bytes) 2024-09-04T21:55:15.014 INFO:teuthology.orchestra.run.smithi112.stdout:4920 read 0x1310d3d thru 0x13201c1 (0xf485 bytes) 2024-09-04T21:55:15.033 INFO:teuthology.orchestra.run.smithi112.stdout:4921 writesame 0x589229a thru 0x589514e data_size 0x1(0x2eb5 bytes) 2024-09-04T21:55:15.034 INFO:teuthology.orchestra.run.smithi112.stdout:4922 read 0x2dbd609 thru 0x2dc73f3 (0x9deb bytes) 2024-09-04T21:55:15.034 INFO:teuthology.orchestra.run.smithi112.stdout:4924 read 0x4949449 thru 0x4952825 (0x93dd bytes) 2024-09-04T21:55:15.035 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:15.050 INFO:teuthology.orchestra.run.smithi112.stdout:4926 trunc from 0x589514f to 0x73a7e 2024-09-04T21:55:15.076 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:15.072+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:15.104 INFO:teuthology.orchestra.run.smithi112.stdout:4929 trunc from 0x73a7e to 0xb8c7d5f 2024-09-04T21:55:15.115 INFO:teuthology.orchestra.run.smithi112.stdout:4930 read 0xf3e442 thru 0xf4ad45 (0xc904 bytes) 2024-09-04T21:55:15.115 INFO:teuthology.orchestra.run.smithi112.stdout:4932 write 0xe8368 thru 0xf7e1d (0xfab6 bytes) 2024-09-04T21:55:15.130 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:15.130 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:15.136 INFO:teuthology.orchestra.run.smithi112.stdout:4934 write 0xdee5ed1 thru 0xdeeb75a (0x588a bytes) 2024-09-04T21:55:15.143 INFO:teuthology.orchestra.run.smithi112.stdout:4936 write 0x585121f thru 0x5851f4c (0xd2e bytes) 2024-09-04T21:55:15.148 INFO:teuthology.orchestra.run.smithi112.stdout:4937 write 0x3fdeae2 thru 0x3fe5726 (0x6c45 bytes) 2024-09-04T21:55:15.153 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:15.156 INFO:teuthology.orchestra.run.smithi112.stdout:4939 read 0x4d263bc thru 0x4d309bc (0xa601 bytes) 2024-09-04T21:55:15.156 INFO:teuthology.orchestra.run.smithi112.stdout:4940 write 0x4d8fdbe thru 0x4d94cef (0x4f32 bytes) 2024-09-04T21:55:15.162 INFO:teuthology.orchestra.run.smithi112.stdout:4942 write 0xa0ad8df thru 0xa0bbcb7 (0xe3d9 bytes) 2024-09-04T21:55:15.168 INFO:teuthology.orchestra.run.smithi112.stdout:4943 read 0x8cdf70c thru 0x8ce40ce (0x49c3 bytes) 2024-09-04T21:55:15.177 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:15.176+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:15.192 INFO:teuthology.orchestra.run.smithi112.stdout:4944 trunc from 0xdeeb75b to 0x2ba7d5 2024-09-04T21:55:15.234 INFO:teuthology.orchestra.run.smithi112.stdout:4945 trunc from 0x2ba7d5 to 0xd1da0cd 2024-09-04T21:55:15.244 INFO:teuthology.orchestra.run.smithi112.stdout:4949 clone 29 order 19 su 8192 sc 11 2024-09-04T21:55:15.278 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:15.276+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:15.288 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:15.288 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:15.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:15.393 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:15.392+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:15.469 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:15.469 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:15.479 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:15.476+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:15.484 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:15.578 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:55:15.581 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:15.576+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:15.652 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:15.652 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:15.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:15.695 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone28 from 0xd1da0cd (overlap 0x73a7e) to 0xb90c 2024-09-04T21:55:15.696 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:15.692+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:15.781 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:15.780+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:15.798 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:15.798 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:15.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:15.865 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-09-04T21:55:15.882 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:15.876+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:15.982 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:15.980+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:15.987 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:15.987 INFO:teuthology.orchestra.run.smithi112.stdout:4952 read 0x8d725ed thru 0x8d72892 (0x2a6 bytes) 2024-09-04T21:55:15.995 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:15.995 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:16.006 INFO:teuthology.orchestra.run.smithi112.stdout:4953 write 0xe860938 thru 0xe862539 (0x1c02 bytes) 2024-09-04T21:55:16.012 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:16.012 INFO:teuthology.orchestra.run.smithi112.stdout:4954 write 0xd2d0faf thru 0xd2d98d5 (0x8927 bytes) 2024-09-04T21:55:16.023 INFO:teuthology.orchestra.run.smithi112.stdout:4955 write 0xd0126eb thru 0xd0186c8 (0x5fde bytes) 2024-09-04T21:55:16.031 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.032 INFO:teuthology.orchestra.run.smithi112.stdout:4957 writesame 0x85d5cf1 thru 0x85e46fd data_size 0xd(0xea0d bytes) 2024-09-04T21:55:16.048 INFO:teuthology.orchestra.run.smithi112.stdout:4958 write 0xb215949 thru 0xb220d5d (0xb415 bytes) 2024-09-04T21:55:16.057 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.057 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.057 INFO:teuthology.orchestra.run.smithi112.stdout:4961 read 0x1d78fe1 thru 0x1d81eea (0x8f0a bytes) 2024-09-04T21:55:16.057 INFO:teuthology.orchestra.run.smithi112.stdout:4962 write 0xbc11e3f thru 0xbc21271 (0xf433 bytes) 2024-09-04T21:55:16.068 INFO:teuthology.orchestra.run.smithi112.stdout:4964 read 0x824f096 thru 0x8250d59 (0x1cc4 bytes) 2024-09-04T21:55:16.068 INFO:teuthology.orchestra.run.smithi112.stdout:4965 punch from 0xcc68f96 to 0xcc7767c, (0xe6e6 bytes) 2024-09-04T21:55:16.068 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.068 INFO:teuthology.orchestra.run.smithi112.stdout:4968 read 0x9b34114 thru 0x9b3e2bd (0xa1aa bytes) 2024-09-04T21:55:16.069 INFO:teuthology.orchestra.run.smithi112.stdout:4969 read 0xab33220 thru 0xab42310 (0xf0f1 bytes) 2024-09-04T21:55:16.069 INFO:teuthology.orchestra.run.smithi112.stdout:4970 write 0x9db66cd thru 0x9dc38b6 (0xd1ea bytes) 2024-09-04T21:55:16.079 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.079 INFO:teuthology.orchestra.run.smithi112.stdout:4972 writesame 0x232a803 thru 0x2334c74 data_size 0x3e(0xa472 bytes) 2024-09-04T21:55:16.083 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:16.080+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:16.088 INFO:teuthology.orchestra.run.smithi112.stdout:4973 write 0x34ed785 thru 0x34ef0e0 (0x195c bytes) 2024-09-04T21:55:16.100 INFO:teuthology.orchestra.run.smithi112.stdout:4975 trunc from 0xe86253a to 0x8ac5874 2024-09-04T21:55:16.131 INFO:teuthology.orchestra.run.smithi112.stdout:4976 write 0x9d9ed6 thru 0x9e2546 (0x8671 bytes) 2024-09-04T21:55:16.139 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.139 INFO:teuthology.orchestra.run.smithi112.stdout:4978 read 0x52cf506 thru 0x52da11c (0xac17 bytes) 2024-09-04T21:55:16.139 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.139 INFO:teuthology.orchestra.run.smithi112.stdout:4981 read 0x337d971 thru 0x3387708 (0x9d98 bytes) 2024-09-04T21:55:16.139 INFO:teuthology.orchestra.run.smithi112.stdout:4983 read 0x5b6f8d9 thru 0x5b7d3f0 (0xdb18 bytes) 2024-09-04T21:55:16.139 INFO:teuthology.orchestra.run.smithi112.stdout:4984 read 0x353b4ae thru 0x3548e0b (0xd95e bytes) 2024-09-04T21:55:16.140 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.140 INFO:teuthology.orchestra.run.smithi112.stdout:4986 write 0x64982b9 thru 0x64a7d50 (0xfa98 bytes) 2024-09-04T21:55:16.150 INFO:teuthology.orchestra.run.smithi112.stdout:4988 writesame 0x27544ef thru 0x2761950 data_size 0x5(0xd462 bytes) 2024-09-04T21:55:16.159 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:16.159 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:16.161 INFO:teuthology.orchestra.run.smithi112.stdout:4989 read 0x4ffae17 thru 0x4ffc4de (0x16c8 bytes) 2024-09-04T21:55:16.162 INFO:teuthology.orchestra.run.smithi112.stdout:4990 writesame 0xca6fa9 thru 0xcb33a9 data_size 0x1(0xc401 bytes) 2024-09-04T21:55:16.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:16.181 INFO:teuthology.orchestra.run.smithi112.stdout:4991 punch from 0x8380367 to 0x83883cc, (0x8065 bytes) 2024-09-04T21:55:16.181 INFO:teuthology.orchestra.run.smithi112.stdout:4992 read 0x5378d55 thru 0x537f35f (0x660b bytes) 2024-09-04T21:55:16.181 INFO:teuthology.orchestra.run.smithi112.stdout:4993 write 0x68e98f3 thru 0x68eba6a (0x2178 bytes) 2024-09-04T21:55:16.184 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:16.180+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:16.185 INFO:teuthology.orchestra.run.smithi112.stdout:4994 punch from 0x80844fb to 0x808c6b5, (0x81ba bytes) 2024-09-04T21:55:16.192 INFO:teuthology.orchestra.run.smithi157.stdout: marking collection for removal 2024-09-04T21:55:16.192 INFO:teuthology.orchestra.run.smithi157.stdout:setting '_remove' omap key 2024-09-04T21:55:16.192 INFO:teuthology.orchestra.run.smithi157.stdout:finish_remove_pgs 3.14_head removing 3.14 2024-09-04T21:55:16.193 INFO:teuthology.orchestra.run.smithi112.stdout:4995 write 0xaad0654 thru 0xaada1c4 (0x9b71 bytes) 2024-09-04T21:55:16.196 INFO:teuthology.orchestra.run.smithi157.stdout:Remove successful 2024-09-04T21:55:16.200 INFO:teuthology.orchestra.run.smithi112.stdout:4997 writesame 0x646c6f5 thru 0x6479276 data_size 0x39(0xcb82 bytes) 2024-09-04T21:55:16.203 INFO:teuthology.orchestra.run.smithi112.stdout:4998 read 0x4fd2432 thru 0x4fd2fb8 (0xb87 bytes) 2024-09-04T21:55:16.204 INFO:teuthology.orchestra.run.smithi112.stdout:5000 punch from 0x5916284 to 0x5924826, (0xe5a2 bytes) 2024-09-04T21:55:16.204 INFO:teuthology.orchestra.run.smithi112.stdout:5001 write 0x6472756 thru 0x6478253 (0x5afe bytes) 2024-09-04T21:55:16.208 INFO:teuthology.orchestra.run.smithi112.stdout:5002 read 0x17d23a2 thru 0x17d88cf (0x652e bytes) 2024-09-04T21:55:16.208 INFO:teuthology.orchestra.run.smithi112.stdout:5004 read 0x3c47e0f thru 0x3c4fc8e (0x7e80 bytes) 2024-09-04T21:55:16.208 INFO:teuthology.orchestra.run.smithi112.stdout:5008 writesame 0x2b3b29d thru 0x2b47626 data_size 0x67(0xc38a bytes) 2024-09-04T21:55:16.229 INFO:teuthology.orchestra.run.smithi112.stdout:5009 trunc from 0xaada1c5 to 0x1525f3d 2024-09-04T21:55:16.267 INFO:teuthology.orchestra.run.smithi112.stdout:5010 read 0x21d273 thru 0x22b2f4 (0xe082 bytes) 2024-09-04T21:55:16.269 INFO:teuthology.orchestra.run.smithi112.stdout:5011 punch from 0x13f6973 to 0x13fa0ca, (0x3757 bytes) 2024-09-04T21:55:16.282 INFO:teuthology.orchestra.run.smithi112.stdout:5012 write 0x51b9fa1 thru 0x51c123d (0x729d bytes) 2024-09-04T21:55:16.284 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:16.280+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:16.290 INFO:teuthology.orchestra.run.smithi112.stdout:5013 punch from 0x1910fe to 0x19915a, (0x805c bytes) 2024-09-04T21:55:16.290 INFO:teuthology.orchestra.run.smithi112.stdout:5015 read 0x4d8dcc6 thru 0x4d8eb09 (0xe44 bytes) 2024-09-04T21:55:16.300 INFO:teuthology.orchestra.run.smithi112.stdout:5016 write 0x680092c thru 0x6806251 (0x5926 bytes) 2024-09-04T21:55:16.323 INFO:teuthology.orchestra.run.smithi112.stdout:5021 write 0xae687d9 thru 0xae77c7e (0xf4a6 bytes) 2024-09-04T21:55:16.324 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:16.324 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:16.337 INFO:teuthology.orchestra.run.smithi112.stdout:5023 read 0xf9db0c thru 0xf9e205 (0x6fa bytes) 2024-09-04T21:55:16.338 INFO:teuthology.orchestra.run.smithi112.stdout:5024 write 0x29d66fe thru 0x29de629 (0x7f2c bytes) 2024-09-04T21:55:16.342 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:16.349 INFO:teuthology.orchestra.run.smithi112.stdout:5025 punch from 0x8611a2 to 0x870dac, (0xfc0a bytes) 2024-09-04T21:55:16.349 INFO:teuthology.orchestra.run.smithi112.stdout:5026 punch from 0x7e095c3 to 0x7e0a38c, (0xdc9 bytes) 2024-09-04T21:55:16.349 INFO:teuthology.orchestra.run.smithi112.stdout:5027 write 0x60b202d thru 0x60bb78b (0x975f bytes) 2024-09-04T21:55:16.361 INFO:teuthology.orchestra.run.smithi112.stdout:5028 write 0x20e5851 thru 0x20f14b6 (0xbc66 bytes) 2024-09-04T21:55:16.373 INFO:teuthology.orchestra.run.smithi112.stdout:5029 punch from 0x827c70c to 0x828c494, (0xfd88 bytes) 2024-09-04T21:55:16.373 INFO:teuthology.orchestra.run.smithi112.stdout:5031 punch from 0x834ab23 to 0x834f1d5, (0x46b2 bytes) 2024-09-04T21:55:16.373 INFO:teuthology.orchestra.run.smithi112.stdout:5033 writesame 0xf2079b thru 0xf2d7bc data_size 0x1(0xd022 bytes) 2024-09-04T21:55:16.385 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:16.384+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:16.390 INFO:teuthology.orchestra.run.smithi112.stdout:5035 write 0x837233a thru 0x837751d (0x51e4 bytes) 2024-09-04T21:55:16.398 INFO:teuthology.orchestra.run.smithi112.stdout:5036 read 0x743e34c thru 0x74400d5 (0x1d8a bytes) 2024-09-04T21:55:16.398 INFO:teuthology.orchestra.run.smithi112.stdout:5037 writesame 0x6068e92 thru 0x606f06c data_size 0xd(0x61db bytes) 2024-09-04T21:55:16.406 INFO:teuthology.orchestra.run.smithi112.stdout:5038 write 0x75d50cf thru 0x75e3c8c (0xebbe bytes) 2024-09-04T21:55:16.420 INFO:teuthology.orchestra.run.smithi112.stdout:5039 punch from 0xc7e3e7 to 0xc8b354, (0xcf6d bytes) 2024-09-04T21:55:16.420 INFO:teuthology.orchestra.run.smithi112.stdout:5040 punch from 0xaa08b3b to 0xaa09ea1, (0x1366 bytes) 2024-09-04T21:55:16.420 INFO:teuthology.orchestra.run.smithi112.stdout:5042 read 0x871e4e4 thru 0x87286fc (0xa219 bytes) 2024-09-04T21:55:16.420 INFO:teuthology.orchestra.run.smithi112.stdout:5043 write 0x17cb0cd thru 0x17d76d8 (0xc60c bytes) 2024-09-04T21:55:16.431 INFO:teuthology.orchestra.run.smithi112.stdout:5044 write 0x2eb4fe4 thru 0x2eb67c2 (0x17df bytes) 2024-09-04T21:55:16.437 INFO:teuthology.orchestra.run.smithi112.stdout:5048 read 0x8b7341 thru 0x8b916d (0x1e2d bytes) 2024-09-04T21:55:16.437 INFO:teuthology.orchestra.run.smithi112.stdout:5049 write 0xdbce48 thru 0xdc228a (0x5443 bytes) 2024-09-04T21:55:16.443 INFO:teuthology.orchestra.run.smithi112.stdout:5050 read 0x25c250e thru 0x25c97b8 (0x72ab bytes) 2024-09-04T21:55:16.444 INFO:teuthology.orchestra.run.smithi112.stdout:5051 read 0x8cfcfef thru 0x8d0a39f (0xd3b1 bytes) 2024-09-04T21:55:16.450 INFO:teuthology.orchestra.run.smithi112.stdout:5052 trunc from 0xae77c7f to 0x63d1514 2024-09-04T21:55:16.472 INFO:teuthology.orchestra.run.smithi112.stdout:5053 punch from 0x568e75e to 0x569ae95, (0xc737 bytes) 2024-09-04T21:55:16.481 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:16.481 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:16.484 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:16.486 INFO:teuthology.orchestra.run.smithi112.stdout:5054 writesame 0x9c23bb7 thru 0x9c2cf17 data_size 0x1(0x9361 bytes) 2024-09-04T21:55:16.486 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.487 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:16.484+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:16.489 INFO:teuthology.orchestra.run.smithi112.stdout:5057 trunc from 0x9c2cf18 to 0xc64dd17 2024-09-04T21:55:16.496 INFO:teuthology.orchestra.run.smithi112.stdout:5058 read 0x94194a3 thru 0x9427a9b (0xe5f9 bytes) 2024-09-04T21:55:16.497 INFO:teuthology.orchestra.run.smithi112.stdout:5059 write 0x1755990 thru 0x1761353 (0xb9c4 bytes) 2024-09-04T21:55:16.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:16.507 INFO:teuthology.orchestra.run.smithi112.stdout:5060 read 0x69b2c97 thru 0x69b4c0e (0x1f78 bytes) 2024-09-04T21:55:16.515 INFO:teuthology.orchestra.run.smithi112.stdout:5061 writesame 0xcf6b48d thru 0xcf711a6 data_size 0x11(0x5d1a bytes) 2024-09-04T21:55:16.515 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.515 INFO:teuthology.orchestra.run.smithi112.stdout:5063 write 0x6007e16 thru 0x6017b68 (0xfd53 bytes) 2024-09-04T21:55:16.521 INFO:teuthology.orchestra.run.smithi112.stdout:5066 write 0x30131bc thru 0x301c6ac (0x94f1 bytes) 2024-09-04T21:55:16.529 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.529 INFO:teuthology.orchestra.run.smithi112.stdout:5069 writesame 0x979468a thru 0x97a3de0 data_size 0x1(0xf757 bytes) 2024-09-04T21:55:16.543 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:55:16.555 INFO:teuthology.orchestra.run.smithi112.stdout:5071 write 0xb0254ef thru 0xb03308e (0xdba0 bytes) 2024-09-04T21:55:16.572 INFO:teuthology.orchestra.run.smithi112.stdout:5072 writesame 0x86c6903 thru 0x86d3f7a data_size 0x4(0xd678 bytes) 2024-09-04T21:55:16.586 INFO:teuthology.orchestra.run.smithi112.stdout:5073 read 0x763435c thru 0x7636a73 (0x2718 bytes) 2024-09-04T21:55:16.586 INFO:teuthology.orchestra.run.smithi112.stdout:5074 read 0xcdb7fe5 thru 0xcdc2b34 (0xab50 bytes) 2024-09-04T21:55:16.586 INFO:teuthology.orchestra.run.smithi112.stdout:5075 write 0x204fd0c thru 0x205cec4 (0xd1b9 bytes) 2024-09-04T21:55:16.587 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:16.584+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:16.600 INFO:teuthology.orchestra.run.smithi112.stdout:5076 trunc from 0xcf711a7 to 0x810ed15 2024-09-04T21:55:16.622 INFO:teuthology.orchestra.run.smithi112.stdout:5078 read 0x5ab7233 thru 0x5ac5448 (0xe216 bytes) 2024-09-04T21:55:16.622 INFO:teuthology.orchestra.run.smithi112.stdout:5079 read 0x429b0e4 thru 0x429f1e9 (0x4106 bytes) 2024-09-04T21:55:16.630 INFO:teuthology.orchestra.run.smithi112.stdout:5081 trunc from 0x810ed15 to 0xdb2404f 2024-09-04T21:55:16.639 INFO:teuthology.orchestra.run.smithi112.stdout:5083 read 0xbe43fd1 thru 0xbe4af0d (0x6f3d bytes) 2024-09-04T21:55:16.639 INFO:teuthology.orchestra.run.smithi112.stdout:5084 punch from 0xc9ba4f5 to 0xc9c3ebc, (0x99c7 bytes) 2024-09-04T21:55:16.639 INFO:teuthology.orchestra.run.smithi112.stdout:5087 writesame 0x5a2d3dc thru 0x5a2e1e3 data_size 0x4(0xe08 bytes) 2024-09-04T21:55:16.645 INFO:teuthology.orchestra.run.smithi112.stdout:5088 punch from 0xd31a5c to 0xd33971, (0x1f15 bytes) 2024-09-04T21:55:16.645 INFO:teuthology.orchestra.run.smithi112.stdout:5089 write 0xa03cb51 thru 0xa0434b2 (0x6962 bytes) 2024-09-04T21:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout:5090 read 0x495bc31 thru 0x4961f6d (0x633d bytes) 2024-09-04T21:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout:5093 read 0x357161c thru 0x357c041 (0xaa26 bytes) 2024-09-04T21:55:16.654 INFO:teuthology.orchestra.run.smithi112.stdout:5094 write 0x673abab thru 0x674a087 (0xf4dd bytes) 2024-09-04T21:55:16.662 INFO:teuthology.orchestra.run.smithi112.stdout:5097 read 0x13e247b thru 0x13e7840 (0x53c6 bytes) 2024-09-04T21:55:16.662 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:16.663 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:16.663 INFO:teuthology.orchestra.run.smithi112.stdout:5098 write 0x4ca2ae thru 0x4cd3a6 (0x30f9 bytes) 2024-09-04T21:55:16.673 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:16.683 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:16.683 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:16.688 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:16.684+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:16.697 INFO:teuthology.orchestra.run.smithi112.stdout:5099 trunc from 0xdb2404f to 0x7ffc5db 2024-09-04T21:55:16.701 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:16.702 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:16.713 INFO:teuthology.orchestra.run.smithi112.stdout:5100 read 0x7ac24c9 thru 0x7ac649b (0x3fd3 bytes) 2024-09-04T21:55:16.721 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:16.727 INFO:teuthology.orchestra.run.smithi112.stdout:5101 write 0xc30827a thru 0xc30bc7a (0x3a01 bytes) 2024-09-04T21:55:16.733 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.14.4 2024-09-04T21:55:16.734 INFO:teuthology.orchestra.run.smithi112.stdout:5102 write 0x7802f6b thru 0x7807507 (0x459d bytes) 2024-09-04T21:55:16.739 INFO:teuthology.orchestra.run.smithi112.stdout:5103 read 0x80ea5ad thru 0x80ea95f (0x3b3 bytes) 2024-09-04T21:55:16.739 INFO:teuthology.orchestra.run.smithi112.stdout:5104 punch from 0xbe0e051 to 0xbe0e913, (0x8c2 bytes) 2024-09-04T21:55:16.740 INFO:teuthology.orchestra.run.smithi112.stdout:5105 write 0x125edf7 thru 0x12693de (0xa5e8 bytes) 2024-09-04T21:55:16.746 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:16.752 INFO:teuthology.orchestra.run.smithi112.stdout:5106 trunc from 0xc30bc7b to 0x613dce0 2024-09-04T21:55:16.771 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:16.771 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:16.780 INFO:teuthology.orchestra.run.smithi112.stdout:5107 write 0x7ecba29 thru 0x7ed96ce (0xdca6 bytes) 2024-09-04T21:55:16.794 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:16.795 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:16.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:16.804 INFO:teuthology.orchestra.run.smithi112.stdout:5114 write 0xc0fc32c thru 0xc103987 (0x765c bytes) 2024-09-04T21:55:16.819 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:16.816+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:16.820 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:16.820 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:16.820 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:16.820 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:16.821 INFO:teuthology.orchestra.run.smithi112.stdout:5115 writesame 0xe8dbf65 thru 0xe8e5b4b data_size 0x1(0x9be7 bytes) 2024-09-04T21:55:16.821 INFO:teuthology.orchestra.run.smithi112.stdout:5116 write 0xd8135a5 thru 0xd81c1c0 (0x8c1c bytes) 2024-09-04T21:55:16.828 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.828 INFO:teuthology.orchestra.run.smithi112.stdout:5118 punch from 0xdd404b9 to 0xdd45396, (0x4edd bytes) 2024-09-04T21:55:16.828 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.829 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.829 INFO:teuthology.orchestra.run.smithi112.stdout:5121 write 0x39a8613 thru 0x39b4d79 (0xc767 bytes) 2024-09-04T21:55:16.838 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:16.839 INFO:teuthology.orchestra.run.smithi112.stdout:5123 trunc from 0xe8e5b4c to 0xd70e547 2024-09-04T21:55:16.846 INFO:teuthology.orchestra.run.smithi112.stdout:5124 punch from 0x23afa78 to 0x23bab7f, (0xb107 bytes) 2024-09-04T21:55:16.846 INFO:teuthology.orchestra.run.smithi112.stdout:5128 write 0x64b8656 thru 0x64c098b (0x8336 bytes) 2024-09-04T21:55:16.853 INFO:teuthology.orchestra.run.smithi112.stdout:5129 write 0x659e6f8 thru 0x659f213 (0xb1c bytes) 2024-09-04T21:55:16.871 INFO:teuthology.orchestra.run.smithi112.stdout:5132 trunc from 0xd70e547 to 0x340464f 2024-09-04T21:55:16.890 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:16.884+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:16.894 INFO:teuthology.orchestra.run.smithi112.stdout:5133 punch from 0x278dcaa to 0x279bcc0, (0xe016 bytes) 2024-09-04T21:55:16.909 INFO:teuthology.orchestra.run.smithi112.stdout:5135 trunc from 0x340464f to 0xe830cfb 2024-09-04T21:55:16.917 INFO:teuthology.orchestra.run.smithi112.stdout:5138 punch from 0x3cf146a to 0x3cf8d52, (0x78e8 bytes) 2024-09-04T21:55:16.917 INFO:teuthology.orchestra.run.smithi112.stdout:5141 write 0x96a02b9 thru 0x96ae64b (0xe393 bytes) 2024-09-04T21:55:16.936 INFO:teuthology.orchestra.run.smithi112.stdout:5142 write 0x8bcfec2 thru 0x8bddeb7 (0xdff6 bytes) 2024-09-04T21:55:16.951 INFO:teuthology.orchestra.run.smithi112.stdout:5145 writesame 0x69d3103 thru 0x69d4f77 data_size 0x17(0x1e75 bytes) 2024-09-04T21:55:16.959 INFO:teuthology.orchestra.run.smithi112.stdout:5146 writesame 0xefa5ae thru 0xf06f5f data_size 0xb(0xc9b2 bytes) 2024-09-04T21:55:16.965 INFO:teuthology.orchestra.run.smithi112.stdout:5147 write 0x59ec852 thru 0x59f3c48 (0x73f7 bytes) 2024-09-04T21:55:16.969 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:16.969 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:16.969 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:16.970 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:16.970 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:16.975 INFO:teuthology.orchestra.run.smithi112.stdout:5149 read 0xba63906 thru 0xba66cd1 (0x33cc bytes) 2024-09-04T21:55:16.975 INFO:teuthology.orchestra.run.smithi112.stdout:5151 write 0xc281dd5 thru 0xc284180 (0x23ac bytes) 2024-09-04T21:55:16.984 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:16.984 INFO:teuthology.orchestra.run.smithi112.stdout:5153 writesame 0x43d1461 thru 0x43d8c4a data_size 0x1(0x77ea bytes) 2024-09-04T21:55:16.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:16.991 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:16.988+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:17.004 INFO:teuthology.orchestra.run.smithi112.stdout:5154 writesame 0x3c9c9de thru 0x3ca78b7 data_size 0x1(0xaeda bytes) 2024-09-04T21:55:17.028 INFO:teuthology.orchestra.run.smithi112.stdout:5155 read 0x2a9a8ed thru 0x2aa01fa (0x590e bytes) 2024-09-04T21:55:17.029 INFO:teuthology.orchestra.run.smithi112.stdout:5156 write 0x8890a19 thru 0x88921da (0x17c2 bytes) 2024-09-04T21:55:17.044 INFO:teuthology.orchestra.run.smithi112.stdout:5157 write 0xe85f360 thru 0xe860b98 (0x1839 bytes) 2024-09-04T21:55:17.051 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:17.051 INFO:teuthology.orchestra.run.smithi112.stdout:5160 write 0xd9ed740 thru 0xd9f7324 (0x9be5 bytes) 2024-09-04T21:55:17.064 INFO:teuthology.orchestra.run.smithi112.stdout:5161 punch from 0x47780a4 to 0x477e092, (0x5fee bytes) 2024-09-04T21:55:17.064 INFO:teuthology.orchestra.run.smithi112.stdout:5162 writesame 0x78a4779 thru 0x78ae784 data_size 0x2(0xa00c bytes) 2024-09-04T21:55:17.083 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:17.088 INFO:teuthology.orchestra.run.smithi112.stdout:5164 trunc from 0xe860b99 to 0xa1f77e8 2024-09-04T21:55:17.092 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:17.088+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:17.104 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:17.113 INFO:teuthology.orchestra.run.smithi112.stdout:5166 write 0xb4a773b thru 0xb4afb3b (0x8401 bytes) 2024-09-04T21:55:17.122 INFO:teuthology.orchestra.run.smithi112.stdout:5167 read 0x9b81aae thru 0x9b89651 (0x7ba4 bytes) 2024-09-04T21:55:17.122 INFO:teuthology.orchestra.run.smithi112.stdout:5168 clone 30 order 20 su 16384 sc 11 2024-09-04T21:55:17.134 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.630947884000001, 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286171498000002, 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.140 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:17.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286142893000001, 2024-09-04T21:55:17.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:17.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:17.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.149 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286122855999999, 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.150 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286103634, 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:17.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286084843000001, 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286052901000001, 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286024372, 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.170 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:17.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.174 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.286004761000001, 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285986468000001, 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:17.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285963654, 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285944517000001, 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285923851, 2024-09-04T21:55:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:17.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.199 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285893074000001, 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.200 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285873706, 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:17.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285849747, 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:17.211 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:17.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.216 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285802497999999, 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.217 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:17.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.221 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285776819999999, 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.222 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:17.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:17.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285742794000001, 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:17.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:17.232 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.285718393, 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:17.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:17.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:17.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.237 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:17.237 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:17.237 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:17.238 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:17.239 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:17.236+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:17.339 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:17.336+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:17.370 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:17.370 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:17.370 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:17.370 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:17.384 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:17.439 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:17.436+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:17.505 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:17.506 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:17.520 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:17.540 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:17.536+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:17.641 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:17.640+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 52.206635055, 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:55:17.651 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:55:17.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:55:17.657 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 52.022094000000003, 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:55:17.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:55:17.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:17.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43351 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.663 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688798571, 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051338846, 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43351 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4499999999999997e-07 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959605+0000", 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959613+0000", 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8250000000000005e-06 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959618+0000", 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.417e-06 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010746+0000", 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051127924999999998 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010835+0000", 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9444000000000006e-05 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010941+0000", 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105312 2024-09-04T21:55:17.667 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43353 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1269760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688774518999999, 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051544344999999998, 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43353 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:17.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959629+0000", 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959635+0000", 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0430000000000004e-06 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959640+0000", 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0320000000000003e-06 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010976+0000", 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051336248000000001 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011067+0000", 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0826000000000001e-05 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011170+0000", 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103507 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43355 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688752377, 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051760575000000003, 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43355 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.673 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:17.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7800000000000002e-07 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959651+0000", 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959658+0000", 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5730000000000002e-06 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959663+0000", 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011205+0000", 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051541489000000003 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011296+0000", 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1105000000000003e-05 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011409+0000", 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112927 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43358 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688729412000001, 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051974999000000001, 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.677 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43358 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9999999999999998e-07 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959674+0000", 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959682+0000", 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9500000000000001e-06 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959687+0000", 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1420000000000001e-06 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011451+0000", 2024-09-04T21:55:17.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051763862000000001 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011538+0000", 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7322000000000002e-05 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011646+0000", 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010772000000000001 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43361 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688705994999999, 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052194391, 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43361 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.682 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959695+0000", 2024-09-04T21:55:17.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959697+0000", 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.418e-06 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959703+0000", 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9989999999999997e-06 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959708+0000", 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.904e-06 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011687+0000", 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051978506000000001 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011781+0000", 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4301000000000004e-05 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011889+0000", 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107796 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43362 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688685204000002, 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052922181999999998, 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.686 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43362 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959716+0000", 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959718+0000", 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959725+0000", 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4660000000000002e-06 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959730+0000", 2024-09-04T21:55:17.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1019999999999996e-06 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011924+0000", 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052193145000000003 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012015+0000", 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0969000000000003e-05 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012637+0000", 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062277700000000003 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43364 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1351680~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688662801, 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052915188000000002, 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.691 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43364 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3799999999999997e-07 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959740+0000", 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959747+0000", 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9149999999999999e-06 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959754+0000", 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8000000000000001e-06 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012425+0000", 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052670531 2024-09-04T21:55:17.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012525+0000", 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100581 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012653+0000", 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012751700000000001 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43366 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.688639152999997, 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052927226000000001, 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43366 2024-09-04T21:55:17.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959762+0000", 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0399999999999996e-07 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959764+0000", 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3949999999999999e-06 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960370+0000", 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060537999999999998 2024-09-04T21:55:17.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960389+0000", 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9326000000000001e-05 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012680+0000", 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052291232 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012688+0000", 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3890000000000005e-06 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43368 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687980467999999, 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052287419000000002, 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.700 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43368 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960422+0000", 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.84e-06 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960431+0000", 2024-09-04T21:55:17.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960971+0000", 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000539965 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960991+0000", 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0832999999999999e-05 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012703+0000", 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051711615000000002 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012707+0000", 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3499999999999999e-06 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43369 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.704 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687379761999999, 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051701824, 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43369 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961022+0000", 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6139999999999999e-06 2024-09-04T21:55:17.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961030+0000", 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1489999999999994e-06 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961051+0000", 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961065+0000", 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4163e-05 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012718+0000", 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051653533000000001 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012722+0000", 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.708 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43371 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687314700000002, 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051652328999999997, 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.709 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43371 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961087+0000", 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961106+0000", 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9097e-05 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961146+0000", 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0281999999999998e-05 2024-09-04T21:55:17.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961162+0000", 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5993000000000001e-05 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012734+0000", 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051571744000000003 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012738+0000", 2024-09-04T21:55:17.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43373 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687216993, 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051570802999999998, 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43373 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961185+0000", 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.423e-06 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961191+0000", 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9429999999999997e-06 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961210+0000", 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8943000000000001e-05 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961223+0000", 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3506000000000001e-05 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012750+0000", 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051527008999999999 2024-09-04T21:55:17.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012754+0000", 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9790000000000004e-06 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43375 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687157818000003, 2024-09-04T21:55:17.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051526660000000002, 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43375 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961244+0000", 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961249+0000", 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961270+0000", 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0925999999999999e-05 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961283+0000", 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2986000000000001e-05 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012765+0000", 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0514825 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012769+0000", 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8079999999999998e-06 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.721 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43377 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687100583000003, 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051486632999999997, 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43377 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961301+0000", 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961306+0000", 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.69e-06 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961323+0000", 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6976000000000001e-05 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961336+0000", 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2913e-05 2024-09-04T21:55:17.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012781+0000", 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051444996999999999 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012786+0000", 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9340000000000003e-06 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.725 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43379 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.687047937000003, 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051454868000000001, 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43379 2024-09-04T21:55:17.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961354+0000", 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0729999999999999e-06 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961358+0000", 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6099999999999999e-06 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961375+0000", 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7091999999999998e-05 2024-09-04T21:55:17.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961390+0000", 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4803e-05 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012803+0000", 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051413343 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012807+0000", 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9469999999999996e-06 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43381 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.686991509000002, 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051415832000000002, 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.730 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43381 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961410+0000", 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961415+0000", 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.977e-06 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961431+0000", 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6016e-05 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961444+0000", 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2816000000000001e-05 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012821+0000", 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051376809000000002 2024-09-04T21:55:17.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012825+0000", 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0910000000000003e-06 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43383 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.686938380999997, 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051378030999999998, 2024-09-04T21:55:17.734 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43383 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961463+0000", 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961473+0000", 2024-09-04T21:55:17.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961535+0000", 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2787999999999996e-05 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961555+0000", 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9646000000000001e-05 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012836+0000", 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051281032999999997 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012840+0000", 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9840000000000003e-06 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43385 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:17.738 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 37.686810434000002, 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051265590999999999, 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43385 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:17.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961591+0000", 2024-09-04T21:55:17.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961597+0000", 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961615+0000", 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7836e-05 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961628+0000", 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2976000000000001e-05 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012851+0000", 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051223109000000003 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012856+0000", 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7870000000000001e-06 2024-09-04T21:55:17.742 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.743 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.743 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.743 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:17.743 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:17.743 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:17.745 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:17.745 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:17.740+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:17.805 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone29 from 0xb4afb3c (overlap 0x1525f3d) to 0xa92f9e 2024-09-04T21:55:17.845 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:17.840+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:17.935 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:17.935 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:17.936 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:17.936 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:17.946 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:17.940+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:17.950 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:18.047 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:18.044+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:18.113 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:18.114 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:18.114 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:18.114 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:18.114 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:18.128 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:18.149 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:18.144+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:18.241 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-09-04T21:55:18.249 INFO:teuthology.orchestra.run.smithi112.stdout:5170 write 0x6b1f40d thru 0x6b2c472 (0xd066 bytes) 2024-09-04T21:55:18.250 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:18.248+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:18.299 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.935542018, 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:18.300 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:55:18.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:55:18.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49175 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 475136~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502845593000004, 2024-09-04T21:55:18.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021719833000000001, 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49175 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793193+0000", 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4e-07 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793196+0000", 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793206+0000", 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4229999999999994e-06 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793213+0000", 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4370000000000001e-06 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814388+0000", 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021174454999999998 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814604+0000", 2024-09-04T21:55:18.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021587900000000001 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814912+0000", 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030848600000000001 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49290 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 40960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:18.311 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.50281493, 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022433502000000001, 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49290 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793224+0000", 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6599999999999996e-07 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793227+0000", 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793240+0000", 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-05 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793252+0000", 2024-09-04T21:55:18.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2843000000000001e-05 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815053+0000", 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021800657000000001 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815301+0000", 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024764399999999999 2024-09-04T21:55:18.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815656+0000", 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035569200000000001 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.316 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49297 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 106496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502774238000001, 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022960876000000002, 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49297 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:18.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793268+0000", 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4460000000000001e-06 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793277+0000", 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2299999999999997e-06 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793284+0000", 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9530000000000002e-06 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815785+0000", 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022500857999999999 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815996+0000", 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211065 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816224+0000", 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022851699999999999 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49303 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 172032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502744415999999, 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023578159000000001, 2024-09-04T21:55:18.321 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49303 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793294+0000", 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793297+0000", 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9500000000000001e-06 2024-09-04T21:55:18.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793306+0000", 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2140000000000002e-06 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793313+0000", 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816342+0000", 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023028835000000001 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816580+0000", 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023813 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816872+0000", 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029135000000000001 2024-09-04T21:55:18.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49309 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 237568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502673454000004, 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024080300999999998, 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.326 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49309 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793365+0000", 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.054e-06 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793374+0000", 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793388+0000", 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3403e-05 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793396+0000", 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8830000000000006e-06 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816958+0000", 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023561992 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817170+0000", 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021269799999999999 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817445+0000", 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027435799999999999 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.330 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49326 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 303104~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502598532, 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024582155000000001, 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49326 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793440+0000", 2024-09-04T21:55:18.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.003e-06 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793445+0000", 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6840000000000004e-06 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793464+0000", 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9085000000000001e-05 2024-09-04T21:55:18.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793478+0000", 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3716999999999999e-05 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817523+0000", 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024045704000000001 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817738+0000", 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021439899999999999 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818021+0000", 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028356300000000002 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.335 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49332 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502546412999997, 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025083616, 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49332 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793492+0000", 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:55:18.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793496+0000", 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793506+0000", 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9729999999999992e-06 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793516+0000", 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6269999999999995e-06 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818099+0000", 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024583880999999998 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818307+0000", 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020786200000000001 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818575+0000", 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026768400000000002 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49338 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.340 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502512226999997, 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025617277000000001, 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49338 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5600000000000005e-07 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793533+0000", 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1930000000000004e-06 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793542+0000", 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0669999999999998e-06 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793552+0000", 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1290000000000002e-06 2024-09-04T21:55:18.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818661+0000", 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025109056000000001 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818870+0000", 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000208929 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819143+0000", 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027324700000000001 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49344 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.502476242, 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026178261000000001, 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.345 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49344 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793565+0000", 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8930000000000001e-06 2024-09-04T21:55:18.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793574+0000", 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793581+0000", 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8399999999999997e-06 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819234+0000", 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025653441999999999 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819463+0000", 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022879500000000001 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819740+0000", 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027665399999999998 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.349 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51043 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168939070999997, 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022470637000000002, 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.350 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51043 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127100+0000", 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127114+0000", 2024-09-04T21:55:18.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4017e-05 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127128+0000", 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4005000000000001e-05 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127139+0000", 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1007e-05 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148542+0000", 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021403575000000001 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148727+0000", 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018442900000000001 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149569+0000", 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084262400000000004 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.354 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51044 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 262144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168884464999998, 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023138729, 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51044 2024-09-04T21:55:18.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127154+0000", 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127158+0000", 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.172e-06 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127194+0000", 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5222000000000003e-05 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127206+0000", 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2159e-05 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149626+0000", 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022420476000000002 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149779+0000", 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015296299999999999 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150292+0000", 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051277400000000002 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51045 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 360448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168813917000001, 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023446379, 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.359 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51045 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127225+0000", 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127230+0000", 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4110000000000004e-06 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127245+0000", 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4714e-05 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127256+0000", 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0787e-05 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150350+0000", 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023094001999999999 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150502+0000", 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151805 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150670+0000", 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016861 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.363 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51046 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.16876559, 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023742831999999998, 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.364 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51046 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127273+0000", 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.23e-07 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127277+0000", 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9650000000000002e-06 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127291+0000", 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3549e-05 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127302+0000", 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1270000000000001e-05 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150727+0000", 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023424996999999999 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150882+0000", 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015530200000000001 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151015+0000", 2024-09-04T21:55:18.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013292600000000001 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51047 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 557056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168722455999998, 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024031869000000001, 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.369 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51047 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:18.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127316+0000", 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5300000000000003e-07 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127320+0000", 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0749999999999999e-06 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127334+0000", 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3587000000000001e-05 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127344+0000", 2024-09-04T21:55:18.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0461e-05 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151070+0000", 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023725311999999998 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151219+0000", 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014918599999999999 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151347+0000", 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012839500000000001 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51048 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168679978999997, 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024346703000000001, 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.374 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51048 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127359+0000", 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.85e-07 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127375+0000", 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6084000000000001e-05 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127389+0000", 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.412e-05 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127400+0000", 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151402+0000", 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024002787000000001 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151573+0000", 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170941 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151705+0000", 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131161 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.378 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51049 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168610532999999, 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024613729000000001, 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.379 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51049 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127428+0000", 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6700000000000002e-07 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127433+0000", 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0000000000000004e-06 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127447+0000", 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4224e-05 2024-09-04T21:55:18.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127459+0000", 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1112e-05 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151761+0000", 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024302397999999999 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151914+0000", 2024-09-04T21:55:18.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015300399999999999 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152041+0000", 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127024 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51050 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.383 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168564230000001, 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024957087999999999, 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51050 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.384 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:18.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127479+0000", 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7160000000000002e-06 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127494+0000", 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4997000000000001e-05 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127506+0000", 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.168e-05 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152104+0000", 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024597810000000001 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152273+0000", 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016888100000000001 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152431+0000", 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000158097 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51051 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.168516736000001, 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025255452000000001, 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.388 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51051 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127522+0000", 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:55:18.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127527+0000", 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6299999999999997e-06 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127541+0000", 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4225e-05 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127552+0000", 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0971999999999999e-05 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152484+0000", 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024931803999999998 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152641+0000", 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015691699999999999 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152776+0000", 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013595400000000001 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:106971 3.0 e82/38 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head v 82'257, mlcod=81'254)", 2024-09-04T21:55:18.393 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.6662224079999994, 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045768883000000003, 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629817+0000", 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9400000000000001e-06 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629834+0000", 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6246999999999999e-05 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629860+0000", 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6628999999999999e-05 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629873+0000", 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3216e-05 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629999+0000", 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012590799999999999 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.630064+0000", 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4546000000000002e-05 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675498+0000", 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045433922000000002 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675584+0000", 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6475000000000001e-05 2024-09-04T21:55:18.397 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.398 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.398 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.398 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:18.401 INFO:teuthology.orchestra.run.smithi112.stdout:5171 trunc from 0xb4afb3c to 0x189940f 2024-09-04T21:55:18.402 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:18.402 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:18.396+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:18.474 INFO:teuthology.orchestra.run.smithi112.stdout:5173 read 0x7ce52b thru 0x7de057 (0xfb2d bytes) 2024-09-04T21:55:18.475 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.475 INFO:teuthology.orchestra.run.smithi112.stdout:5175 punch from 0xc3214f to 0xc39f2b, (0x7ddc bytes) 2024-09-04T21:55:18.475 INFO:teuthology.orchestra.run.smithi112.stdout:5176 read 0x1309ac3 thru 0x1318d08 (0xf246 bytes) 2024-09-04T21:55:18.476 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.502 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:18.500+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:18.506 INFO:teuthology.orchestra.run.smithi112.stdout:5179 writesame 0xe9bc6dc thru 0xe9c51ce data_size 0x47(0x8af3 bytes) 2024-09-04T21:55:18.513 INFO:teuthology.orchestra.run.smithi112.stdout:5181 trunc from 0xe9c51cf to 0xb1980f4 2024-09-04T21:55:18.522 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.522 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.522 INFO:teuthology.orchestra.run.smithi112.stdout:5186 read 0x38f861c thru 0x38ff7c6 (0x71ab bytes) 2024-09-04T21:55:18.536 INFO:teuthology.orchestra.run.smithi112.stdout:5187 write 0xd7d3949 thru 0xd7d4477 (0xb2f bytes) 2024-09-04T21:55:18.569 INFO:teuthology.orchestra.run.smithi112.stdout:5188 trunc from 0xd7d4478 to 0x918979 2024-09-04T21:55:18.576 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:55:18.579 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:18.579 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:18.579 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:18.579 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:18.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:18.621 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:18.620+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:18.704 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:18.700+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:18.724 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:18.724 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:18.724 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:18.724 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:18.724 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:18.737 INFO:teuthology.orchestra.run.smithi112.stdout:5189 read 0x2e07 thru 0x44d6 (0x16d0 bytes) 2024-09-04T21:55:18.738 INFO:teuthology.orchestra.run.smithi112.stdout:5190 punch from 0x5c1ff to 0x6bded, (0xfbee bytes) 2024-09-04T21:55:18.738 INFO:teuthology.orchestra.run.smithi112.stdout:5191 read 0x608347 thru 0x614ab4 (0xc76e bytes) 2024-09-04T21:55:18.738 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.740 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:18.763 INFO:teuthology.orchestra.run.smithi112.stdout:5193 write 0xc81af99 thru 0xc81dfca (0x3032 bytes) 2024-09-04T21:55:18.768 INFO:teuthology.orchestra.run.smithi112.stdout:5194 read 0x61f50de thru 0x61f56ad (0x5d0 bytes) 2024-09-04T21:55:18.768 INFO:teuthology.orchestra.run.smithi112.stdout:5195 write 0x61ee11f thru 0x61faf58 (0xce3a bytes) 2024-09-04T21:55:18.777 INFO:teuthology.orchestra.run.smithi112.stdout:5196 write 0x944a3a8 thru 0x94502e2 (0x5f3b bytes) 2024-09-04T21:55:18.785 INFO:teuthology.orchestra.run.smithi112.stdout:5197 punch from 0x54458f0 to 0x5453a92, (0xe1a2 bytes) 2024-09-04T21:55:18.785 INFO:teuthology.orchestra.run.smithi112.stdout:5199 write 0x2d5dfb8 thru 0x2d66a3e (0x8a87 bytes) 2024-09-04T21:55:18.793 INFO:teuthology.orchestra.run.smithi112.stdout:5200 read 0x1ee75c7 thru 0x1eee991 (0x73cb bytes) 2024-09-04T21:55:18.793 INFO:teuthology.orchestra.run.smithi112.stdout:5202 read 0x21a3338 thru 0x21acbd0 (0x9899 bytes) 2024-09-04T21:55:18.793 INFO:teuthology.orchestra.run.smithi112.stdout:5204 punch from 0x362c000 to 0x363a6cf, (0xe6cf bytes) 2024-09-04T21:55:18.794 INFO:teuthology.orchestra.run.smithi112.stdout:5206 punch from 0x8bcc0fc to 0x8bd9e12, (0xdd16 bytes) 2024-09-04T21:55:18.794 INFO:teuthology.orchestra.run.smithi112.stdout:5207 punch from 0x11dc6a3 to 0x11de07f, (0x19dc bytes) 2024-09-04T21:55:18.794 INFO:teuthology.orchestra.run.smithi112.stdout:5208 writesame 0xc3feee4 thru 0xc402f3f data_size 0x6(0x405c bytes) 2024-09-04T21:55:18.798 INFO:teuthology.orchestra.run.smithi112.stdout:5209 punch from 0x12343fd to 0x123c3f1, (0x7ff4 bytes) 2024-09-04T21:55:18.798 INFO:teuthology.orchestra.run.smithi112.stdout:5210 read 0x2f02cc9 thru 0x2f0cd2b (0xa063 bytes) 2024-09-04T21:55:18.798 INFO:teuthology.orchestra.run.smithi112.stdout:5211 read 0x77bf9a2 thru 0x77c2f73 (0x35d2 bytes) 2024-09-04T21:55:18.799 INFO:teuthology.orchestra.run.smithi112.stdout:5212 punch from 0x6615371 to 0x6618764, (0x33f3 bytes) 2024-09-04T21:55:18.799 INFO:teuthology.orchestra.run.smithi112.stdout:5213 write 0x8a61301 thru 0x8a636c0 (0x23c0 bytes) 2024-09-04T21:55:18.801 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:18.802 INFO:teuthology.orchestra.run.smithi112.stdout:5214 writesame 0x5b9263e thru 0x5b95a49 data_size 0x2(0x340c bytes) 2024-09-04T21:55:18.804 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:18.800+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:18.805 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.818 INFO:teuthology.orchestra.run.smithi112.stdout:5216 trunc from 0xc81dfcb to 0x433ea6f 2024-09-04T21:55:18.840 INFO:teuthology.orchestra.run.smithi112.stdout:5217 trunc from 0x433ea6f to 0x9770af5 2024-09-04T21:55:18.846 INFO:teuthology.orchestra.run.smithi112.stdout:5218 punch from 0x8689151 to 0x8689d84, (0xc33 bytes) 2024-09-04T21:55:18.846 INFO:teuthology.orchestra.run.smithi112.stdout:5219 read 0x6d0d823 thru 0x6d15047 (0x7825 bytes) 2024-09-04T21:55:18.846 INFO:teuthology.orchestra.run.smithi112.stdout:5220 read 0x591c5b8 thru 0x59268a4 (0xa2ed bytes) 2024-09-04T21:55:18.846 INFO:teuthology.orchestra.run.smithi112.stdout:5221 write 0x87dfaca thru 0x87ef539 (0xfa70 bytes) 2024-09-04T21:55:18.866 INFO:teuthology.orchestra.run.smithi112.stdout:5224 write 0xbcebbed thru 0xbcf3025 (0x7439 bytes) 2024-09-04T21:55:18.869 INFO:teuthology.orchestra.run.smithi112.stdout:5225 read 0x9903ba3 thru 0x99121ad (0xe60b bytes) 2024-09-04T21:55:18.871 INFO:teuthology.orchestra.run.smithi157.stdout:get_pg_num_history pg_num_history pg_num_history(e84 pg_nums {1={14=1},2={18=8},3={22=32}} deleted_pools ) 2024-09-04T21:55:18.871 INFO:teuthology.orchestra.run.smithi157.stdout:Importing pgid 3.14 2024-09-04T21:55:18.872 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:284215d9:::rbd_data.1188e7c968d9.0000000000000002:4# 2024-09-04T21:55:18.873 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:284215d9:::rbd_data.1188e7c968d9.0000000000000002:head# 2024-09-04T21:55:18.873 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 4=[]:{4=[4]} 2024-09-04T21:55:18.874 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2850fa02:::rbd_data.118816ab2dec.000000000000016f:1b# 2024-09-04T21:55:18.875 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2850fa02:::rbd_data.118816ab2dec.000000000000016f:head# 2024-09-04T21:55:18.875 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 1b=[]:{1b=[1b]} 2024-09-04T21:55:18.875 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2891a0a1:::rbd_data.11882296a3bb.0000000000000048:12# 2024-09-04T21:55:18.876 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2891a0a1:::rbd_data.11882296a3bb.0000000000000048:head# 2024-09-04T21:55:18.876 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 12=[]:{12=[12]} 2024-09-04T21:55:18.877 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:297090b0:::rbd_data.1188b450424.0000000000000017:14# 2024-09-04T21:55:18.877 INFO:teuthology.orchestra.run.smithi112.stdout:5227 trunc from 0xbcf3026 to 0x72d13b8 2024-09-04T21:55:18.878 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:297090b0:::rbd_data.1188b450424.0000000000000017:head# 2024-09-04T21:55:18.878 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 14=[]:{14=[14]} 2024-09-04T21:55:18.878 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:29898bb4:::rbd_data.1188b450424.0000000000000018:14# 2024-09-04T21:55:18.879 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:29898bb4:::rbd_data.1188b450424.0000000000000018:head# 2024-09-04T21:55:18.879 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 14=[]:{14=[14]} 2024-09-04T21:55:18.879 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2a299d23:::rbd_data.1188e7c968d9.000000000000007c:4# 2024-09-04T21:55:18.882 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2a299d23:::rbd_data.1188e7c968d9.000000000000007c:head# 2024-09-04T21:55:18.882 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 4=[]:{4=[4]} 2024-09-04T21:55:18.882 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2acf6680:::rbd_object_map.11883c2bd208:head# 2024-09-04T21:55:18.882 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.882 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2af03d74:::rbd_data.1188388ff6d.000000000000006b:head# 2024-09-04T21:55:18.883 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.884 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2afd88fa:::rbd_data.1188c8e6983d.0000000000000000:head# 2024-09-04T21:55:18.884 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.884 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2b5a5038:::rbd_data.1188e5a3a295.0000000000000001:18# 2024-09-04T21:55:18.887 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2b5a5038:::rbd_data.1188e5a3a295.0000000000000001:head# 2024-09-04T21:55:18.888 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 18=[]:{18=[18]} 2024-09-04T21:55:18.888 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2ba9bd77:::rbd_data.118816ab2dec.0000000000000048:1b# 2024-09-04T21:55:18.889 INFO:teuthology.orchestra.run.smithi112.stdout:5228 read 0x30f5444 thru 0x30f93d7 (0x3f94 bytes) 2024-09-04T21:55:18.889 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2ba9bd77:::rbd_data.118816ab2dec.0000000000000048:head# 2024-09-04T21:55:18.889 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 1b=[]:{1b=[1b]} 2024-09-04T21:55:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2bb4acb3:::rbd_id.image_client.0-clone5:head# 2024-09-04T21:55:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2bb82ebc:::rbd_data.11883c2bd208.0000000000000045:head# 2024-09-04T21:55:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 11=[]:{} 2024-09-04T21:55:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2bc09623:::rbd_data.1188b450424.0000000000000055:14# 2024-09-04T21:55:18.890 INFO:teuthology.orchestra.run.smithi112.stdout:5230 read 0x455cf2e thru 0x455dd10 (0xde3 bytes) 2024-09-04T21:55:18.891 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:18.891 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2bc09623:::rbd_data.1188b450424.0000000000000055:head# 2024-09-04T21:55:18.891 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 14=[]:{14=[14]} 2024-09-04T21:55:18.891 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2bc25c91:::rbd_data.1188aa2c625d.0000000000000023:head# 2024-09-04T21:55:18.892 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.894 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2cfeb677:::rbd_data.118816ab2dec.0000000000000171:1b# 2024-09-04T21:55:18.894 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2cfeb677:::rbd_data.118816ab2dec.0000000000000171:head# 2024-09-04T21:55:18.894 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 1b=[]:{1b=[1b]} 2024-09-04T21:55:18.895 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2d87aeef:::rbd_data.1188388ff6d.0000000000000005:head# 2024-09-04T21:55:18.895 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.895 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2de59dee:::rbd_data.11885236e45a.00000000000000e7:head# 2024-09-04T21:55:18.895 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 1d=[]:{} 2024-09-04T21:55:18.896 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2e0da751:::rbd_object_map.1188c8e6983d.0000000000000017:head# 2024-09-04T21:55:18.896 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 0=[]:{} 2024-09-04T21:55:18.896 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2faad357:::rbd_data.1188e5a3a295.0000000000000002:18# 2024-09-04T21:55:18.897 INFO:teuthology.orchestra.run.smithi157.stdout:Write #3:2faad357:::rbd_data.1188e5a3a295.0000000000000002:head# 2024-09-04T21:55:18.897 INFO:teuthology.orchestra.run.smithi157.stdout:snapset 18=[]:{18=[18]} 2024-09-04T21:55:18.898 INFO:teuthology.orchestra.run.smithi157.stdout:write_pg epoch 84 info 3.14( v 84'164 (0'0,84'164] local-lis/les=48/49 n=31 ec=22/22 lis/c=48/48 les/c/f=49/49/0 sis=48) 2024-09-04T21:55:18.899 INFO:teuthology.orchestra.run.smithi112.stdout:5234 write 0x954e04e thru 0x95552c2 (0x7275 bytes) 2024-09-04T21:55:18.899 INFO:teuthology.orchestra.run.smithi157.stdout:Import successful 2024-09-04T21:55:18.905 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:18.904+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:18.912 INFO:teuthology.orchestra.run.smithi112.stdout:5236 trunc from 0x95552c3 to 0xcc1ed93 2024-09-04T21:55:18.919 INFO:teuthology.orchestra.run.smithi112.stdout:5237 write 0xcfc49 thru 0xd9101 (0x94b9 bytes) 2024-09-04T21:55:18.921 INFO:teuthology.orchestra.run.smithi112.stdout:5238 writesame 0x90d2be2 thru 0x90e1c69 data_size 0x56(0xf088 bytes) 2024-09-04T21:55:18.927 INFO:teuthology.orchestra.run.smithi112.stdout:5240 punch from 0x893690c to 0x8945420, (0xeb14 bytes) 2024-09-04T21:55:18.927 INFO:teuthology.orchestra.run.smithi112.stdout:5241 read 0xb0a7fec thru 0xb0b6841 (0xe856 bytes) 2024-09-04T21:55:18.927 INFO:teuthology.orchestra.run.smithi112.stdout:5242 read 0x113de28 thru 0x1143212 (0x53eb bytes) 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 48.310358004999998, 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:18.928 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:18.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:18.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 48.299352761999998, 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:18.933 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:18.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755669988000001, 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.938 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:55:18.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755646012, 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.943 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755605326000001, 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.947 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:18.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:18.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.951 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755571926999998, 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:18.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:18.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755528656000003, 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:18.956 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:18.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.960 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755483974999997, 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.961 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:18.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755422819000003, 2024-09-04T21:55:18.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:18.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.969 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755392778000001, 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.970 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.974 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755348007000002, 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:18.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755299715, 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755264070000003, 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.984 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:18.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:18.988 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.755014854000002, 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:18.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.992 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.754975985000002, 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.993 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:18.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:18.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.749829239999997, 2024-09-04T21:55:18.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:18.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:18.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:19.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.749785981000002, 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:19.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:19.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.006 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.74974211, 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:19.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:19.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:19.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.749705831999997, 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:19.011 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:19.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:19.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.015 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.749679725999997, 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:19.016 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:19.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:19.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:19.020 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:19.021 INFO:teuthology.orchestra.run.smithi112.stdout:5243 trunc from 0xcc1ed93 to 0x6535d21 2024-09-04T21:55:19.022 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:19.022 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:19.022 INFO:teuthology.orchestra.run.smithi112.stdout:5244 writesame 0xc9c690c thru 0xc9d0857 data_size 0xa(0x9f4c bytes) 2024-09-04T21:55:19.023 INFO:teuthology.orchestra.run.smithi112.stdout:5245 trunc from 0xc9d0858 to 0x9bc67fe 2024-09-04T21:55:19.023 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:19.024 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:19.024 INFO:teuthology.orchestra.run.smithi112.stdout:5246 read 0x37f9860 thru 0x3809141 (0xf8e2 bytes) 2024-09-04T21:55:19.024 INFO:teuthology.orchestra.run.smithi112.stdout:5247 read 0x99db545 thru 0x99df729 (0x41e5 bytes) 2024-09-04T21:55:19.025 INFO:teuthology.orchestra.run.smithi112.stdout:5248 read 0x8a52048 thru 0x8a58898 (0x6851 bytes) 2024-09-04T21:55:19.025 INFO:teuthology.orchestra.run.smithi112.stdout:5249 read 0x4fed7ef thru 0x4ffccaa (0xf4bc bytes) 2024-09-04T21:55:19.025 INFO:teuthology.orchestra.run.smithi112.stdout:5250 write 0xeacdd4a thru 0xeadbd22 (0xdfd9 bytes) 2024-09-04T21:55:19.025 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:19.025 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:19.026 INFO:teuthology.orchestra.run.smithi112.stdout:5251 trunc from 0xeadbd23 to 0xd886450 2024-09-04T21:55:19.026 INFO:teuthology.orchestra.run.smithi112.stdout:5252 write 0x1ab9b8 thru 0x1b0525 (0x4b6e bytes) 2024-09-04T21:55:19.026 INFO:teuthology.orchestra.run.smithi112.stdout:5253 writesame 0x99062ba thru 0x990f048 data_size 0x1f(0x8d8f bytes) 2024-09-04T21:55:19.026 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:19.026 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:19.027 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:19.027 INFO:teuthology.orchestra.run.smithi112.stdout:5255 punch from 0xb506e3a to 0xb50a944, (0x3b0a bytes) 2024-09-04T21:55:19.027 INFO:teuthology.orchestra.run.smithi112.stdout:5256 write 0xe3f0b80 thru 0xe3fe5bc (0xda3d bytes) 2024-09-04T21:55:19.027 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:19.020+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:19.028 INFO:teuthology.orchestra.run.smithi112.stdout:5258 trunc from 0xe3fe5bd to 0xd793aa6 2024-09-04T21:55:19.028 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:19.028 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:19.031 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.045 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:19.047 INFO:teuthology.orchestra.run.smithi112.stdout:5261 trunc from 0xd793aa6 to 0x2e738f5 2024-09-04T21:55:19.060 INFO:teuthology.orchestra.run.smithi112.stdout:5264 read 0x19db5f6 thru 0x19e6f3f (0xb94a bytes) 2024-09-04T21:55:19.060 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.060 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.066 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:19.066 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:19.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:19.074 INFO:teuthology.orchestra.run.smithi112.stdout:5270 write 0x6a0063b thru 0x6a04c9e (0x4664 bytes) 2024-09-04T21:55:19.081 INFO:teuthology.orchestra.run.smithi112.stdout:5271 writesame 0x3e45a77 thru 0x3e4fb95 data_size 0x3(0xa11f bytes) 2024-09-04T21:55:19.099 INFO:teuthology.orchestra.run.smithi112.stdout:5272 trunc from 0x6a04c9f to 0xb3853ca 2024-09-04T21:55:19.112 INFO:teuthology.orchestra.run.smithi112.stdout:5273 writesame 0xae9a6eb thru 0xae9b91d data_size 0x1(0x1233 bytes) 2024-09-04T21:55:19.119 INFO:teuthology.orchestra.run.smithi112.stdout:5277 read 0xa90be66 thru 0xa91240f (0x65aa bytes) 2024-09-04T21:55:19.119 INFO:teuthology.orchestra.run.smithi112.stdout:5278 read 0x32b993a thru 0x32baf10 (0x15d7 bytes) 2024-09-04T21:55:19.120 INFO:teuthology.orchestra.run.smithi112.stdout:5279 read 0x8703604 thru 0x8703748 (0x145 bytes) 2024-09-04T21:55:19.120 INFO:teuthology.orchestra.run.smithi112.stdout:5280 read 0x3379d48 thru 0x337ee0b (0x50c4 bytes) 2024-09-04T21:55:19.122 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:19.120+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:19.126 INFO:teuthology.orchestra.run.smithi112.stdout:5281 trunc from 0xb3853ca to 0x75d870a 2024-09-04T21:55:19.147 INFO:teuthology.orchestra.run.smithi112.stdout:5283 writesame 0xe0af52 thru 0xe1ab5c data_size 0x1(0xfc0b bytes) 2024-09-04T21:55:19.150 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:19.150 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:19.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:19.197 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.208 INFO:teuthology.orchestra.run.smithi112.stdout:5287 trunc from 0x75d870a to 0xbcdd99 2024-09-04T21:55:19.223 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:19.220+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:19.230 INFO:teuthology.orchestra.run.smithi112.stdout:5289 punch from 0x9fc169 to 0xa00a16, (0x48ad bytes) 2024-09-04T21:55:19.230 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.234 INFO:teuthology.orchestra.run.smithi112.stdout:5292 trunc from 0xbcdd99 to 0x37e82e2 2024-09-04T21:55:19.242 INFO:teuthology.orchestra.run.smithi112.stdout:5293 punch from 0x5327e9 to 0x533589, (0xda0 bytes) 2024-09-04T21:55:19.242 INFO:teuthology.orchestra.run.smithi112.stdout:5294 read 0xc7ddc1 thru 0xc8a911 (0xcb51 bytes) 2024-09-04T21:55:19.243 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.269 INFO:teuthology.orchestra.run.smithi112.stdout:5296 write 0xd44765c thru 0xd44ff83 (0x8928 bytes) 2024-09-04T21:55:19.278 INFO:teuthology.orchestra.run.smithi112.stdout:5297 write 0x5a07522 thru 0x5a0da50 (0x652f bytes) 2024-09-04T21:55:19.289 INFO:teuthology.orchestra.run.smithi112.stdout:5298 write 0x4af07ae thru 0x4af10d9 (0x92c bytes) 2024-09-04T21:55:19.295 INFO:teuthology.orchestra.run.smithi112.stdout:5299 write 0xca8791d thru 0xca8ec29 (0x730d bytes) 2024-09-04T21:55:19.305 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.305 INFO:teuthology.orchestra.run.smithi112.stdout:5303 read 0xe54f1f thru 0xe5bbb8 (0x6c9a bytes) 2024-09-04T21:55:19.313 INFO:teuthology.orchestra.run.smithi112.stdout:5305 trunc from 0xd44ff84 to 0x846a9b5 2024-09-04T21:55:19.324 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:19.320+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:19.324 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:19.324 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:19.339 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:19.341 INFO:teuthology.orchestra.run.smithi112.stdout:5308 trunc from 0x846a9b5 to 0x53fb484 2024-09-04T21:55:19.364 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.364 INFO:teuthology.orchestra.run.smithi112.stdout:5311 flatten 2024-09-04T21:55:19.379 INFO:teuthology.orchestra.run.smithi112.stdout:5312 write 0x7d8ce5 thru 0x7e48df (0xbbfb bytes) 2024-09-04T21:55:19.395 INFO:teuthology.orchestra.run.smithi112.stdout:5313 trunc from 0x53fb484 to 0xb659aba 2024-09-04T21:55:19.418 INFO:teuthology.orchestra.run.smithi112.stdout:5314 write 0xb9c4570 thru 0xb9d0b92 (0xc623 bytes) 2024-09-04T21:55:19.425 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:19.424+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:19.429 INFO:teuthology.orchestra.run.smithi112.stdout:5318 writesame 0xb9b273d thru 0xb9ba4fa data_size 0x57(0x7dbe bytes) 2024-09-04T21:55:19.436 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /var/log/ceph/exp.3.14.4 2024-09-04T21:55:19.439 INFO:teuthology.orchestra.run.smithi112.stdout:5320 read 0x27e0f3c thru 0x27ebfae (0xb073 bytes) 2024-09-04T21:55:19.439 INFO:teuthology.orchestra.run.smithi112.stdout:5321 write 0x9a2950c thru 0x9a2be51 (0x2946 bytes) 2024-09-04T21:55:19.457 INFO:teuthology.orchestra.run.smithi112.stdout:5323 writesame 0xe3b3261 thru 0xe3bcd6f data_size 0x11(0x9b0f bytes) 2024-09-04T21:55:19.457 INFO:teuthology.orchestra.run.smithi112.stdout:5326 write 0x4512fc8 thru 0x451e97c (0xb9b5 bytes) 2024-09-04T21:55:19.463 INFO:teuthology.orchestra.run.smithi112.stdout:5327 read 0x83eece3 thru 0x83f16cf (0x29ed bytes) 2024-09-04T21:55:19.468 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:19.469 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:19.479 INFO:teuthology.orchestra.run.smithi112.stdout:5328 trunc from 0xe3bcd70 to 0x3d71c6e 2024-09-04T21:55:19.492 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:19.497 INFO:teuthology.orchestra.run.smithi112.stdout:5329 read 0xf0b3bf thru 0xf106e6 (0x5328 bytes) 2024-09-04T21:55:19.498 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.498 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.514 INFO:teuthology.orchestra.run.smithi112.stdout:5335 write 0x7b7d6ee thru 0x7b7f294 (0x1ba7 bytes) 2024-09-04T21:55:19.520 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.520 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.520 INFO:teuthology.orchestra.run.smithi112.stdout:5340 read 0x439564d thru 0x4399175 (0x3b29 bytes) 2024-09-04T21:55:19.520 INFO:teuthology.orchestra.run.smithi112.stdout:5341 read 0x400fd7c thru 0x4018f10 (0x9195 bytes) 2024-09-04T21:55:19.520 INFO:teuthology.orchestra.run.smithi112.stdout:5342 read 0x5c4ba9b thru 0x5c4dfa5 (0x250b bytes) 2024-09-04T21:55:19.526 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:19.520+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:19.530 INFO:teuthology.orchestra.run.smithi112.stdout:5343 writesame 0x890bb0e thru 0x8914d67 data_size 0x1a(0x925a bytes) 2024-09-04T21:55:19.531 INFO:teuthology.orchestra.run.smithi112.stdout:5344 read 0x3c96e69 thru 0x3c98db5 (0x1f4d bytes) 2024-09-04T21:55:19.531 INFO:teuthology.orchestra.run.smithi112.stdout:5345 read 0x72b0fea thru 0x72b6ada (0x5af1 bytes) 2024-09-04T21:55:19.531 INFO:teuthology.orchestra.run.smithi112.stdout:5347 read 0x1fbef3f thru 0x1fc5f75 (0x7037 bytes) 2024-09-04T21:55:19.531 INFO:teuthology.orchestra.run.smithi112.stdout:5348 write 0x74ed6a8 thru 0x74f07a3 (0x30fc bytes) 2024-09-04T21:55:19.537 INFO:teuthology.orchestra.run.smithi112.stdout:5349 writesame 0x4f56da4 thru 0x4f5784c data_size 0x1(0xaa9 bytes) 2024-09-04T21:55:19.556 INFO:teuthology.orchestra.run.smithi112.stdout:5352 trunc from 0x8914d68 to 0x1974f0b 2024-09-04T21:55:19.570 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:55:19.607 INFO:teuthology.orchestra.run.smithi112.stdout:5353 writesame 0xc539396 thru 0xc53bcb3 data_size 0x2(0x291e bytes) 2024-09-04T21:55:19.608 INFO:teuthology.orchestra.run.smithi112.stdout:5355 punch from 0x5a38744 to 0x5a4762b, (0xeee7 bytes) 2024-09-04T21:55:19.608 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.608 INFO:teuthology.orchestra.run.smithi112.stdout:5358 punch from 0x6bf7db4 to 0x6bfd061, (0x52ad bytes) 2024-09-04T21:55:19.608 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.610 INFO:teuthology.orchestra.run.smithi112.stdout:5363 trunc from 0xc53bcb4 to 0xec579eb 2024-09-04T21:55:19.619 INFO:teuthology.orchestra.run.smithi112.stdout:5364 read 0x8e4e207 thru 0x8e5bc69 (0xda63 bytes) 2024-09-04T21:55:19.619 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.619 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.627 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:19.624+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:19.633 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:19.634 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:19.635 INFO:teuthology.orchestra.run.smithi112.stdout:5368 trunc from 0xec579eb to 0x4215256 2024-09-04T21:55:19.645 INFO:teuthology.orchestra.run.smithi112.stdout:5369 punch from 0x373a8a9 to 0x373b611, (0xd68 bytes) 2024-09-04T21:55:19.645 INFO:teuthology.orchestra.run.smithi112.stdout:5370 read 0x1489ffc thru 0x1496ee0 (0xcee5 bytes) 2024-09-04T21:55:19.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:19.662 INFO:teuthology.orchestra.run.smithi112.stdout:5372 write 0x4b9f9f7 thru 0x4ba8f6d (0x9577 bytes) 2024-09-04T21:55:19.699 INFO:teuthology.orchestra.run.smithi112.stdout:5374 write 0xbbefcf2 thru 0xbbfa4b3 (0xa7c2 bytes) 2024-09-04T21:55:19.711 INFO:teuthology.orchestra.run.smithi112.stdout:5375 read 0x975d7b4 thru 0x97693c1 (0xbc0e bytes) 2024-09-04T21:55:19.724 INFO:teuthology.orchestra.run.smithi112.stdout:5378 trunc from 0xbbfa4b4 to 0x3187a6a 2024-09-04T21:55:19.728 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:19.724+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:19.747 INFO:teuthology.orchestra.run.smithi112.stdout:5379 write 0x6e6e27 thru 0x6f2ac1 (0xbc9b bytes) 2024-09-04T21:55:19.752 INFO:teuthology.orchestra.run.smithi112.stdout:5380 write 0x246248d thru 0x2465d2f (0x38a3 bytes) 2024-09-04T21:55:19.761 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.761 INFO:teuthology.orchestra.run.smithi112.stdout:5383 punch from 0x908e28 to 0x90a7e5, (0x19bd bytes) 2024-09-04T21:55:19.779 INFO:teuthology.orchestra.run.smithi112.stdout:5384 trunc from 0x3187a6a to 0xdefb24e 2024-09-04T21:55:19.788 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:19.788 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:19.792 INFO:teuthology.orchestra.run.smithi112.stdout:5385 punch from 0x6ec2323 to 0x6ecc85d, (0xa53a bytes) 2024-09-04T21:55:19.792 INFO:teuthology.orchestra.run.smithi112.stdout:5386 punch from 0x18c012d to 0x18c1ef2, (0x1dc5 bytes) 2024-09-04T21:55:19.792 INFO:teuthology.orchestra.run.smithi112.stdout:5388 write 0x3b7784f thru 0x3b7d724 (0x5ed6 bytes) 2024-09-04T21:55:19.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:19.825 INFO:teuthology.orchestra.run.smithi112.stdout:5389 trunc from 0xdefb24e to 0x26acaa3 2024-09-04T21:55:19.869 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:19.868+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:19.879 INFO:teuthology.orchestra.run.smithi112.stdout:5391 write 0xc119057 thru 0xc121caa (0x8c54 bytes) 2024-09-04T21:55:19.891 INFO:teuthology.orchestra.run.smithi112.stdout:5393 read 0x8a3038b thru 0x8a31507 (0x117d bytes) 2024-09-04T21:55:19.891 INFO:teuthology.orchestra.run.smithi112.stdout:5394 read 0x90d73b5 thru 0x90e0979 (0x95c5 bytes) 2024-09-04T21:55:19.891 INFO:teuthology.orchestra.run.smithi112.stdout:5395 read 0xa4e9a29 thru 0xa4eed80 (0x5358 bytes) 2024-09-04T21:55:19.891 INFO:teuthology.orchestra.run.smithi112.stdout:5396 write 0x2f798a7 thru 0x2f7b24c (0x19a6 bytes) 2024-09-04T21:55:19.897 INFO:teuthology.orchestra.run.smithi112.stdout:5399 writesame 0xa26edb1 thru 0xa27a154 data_size 0x2(0xb3a4 bytes) 2024-09-04T21:55:19.923 INFO:teuthology.orchestra.run.smithi112.stdout:truncating to largest ever: 0xee298e1 2024-09-04T21:55:19.927 INFO:teuthology.orchestra.run.smithi112.stdout:5400 trunc from 0xc121cab to 0xee298e1 2024-09-04T21:55:19.928 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:19.924+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:19.941 INFO:teuthology.orchestra.run.smithi112.stdout:5403 writesame 0x593a5f1 thru 0x593b2af data_size 0x1(0xcbf bytes) 2024-09-04T21:55:19.948 INFO:teuthology.orchestra.run.smithi112.stdout:5404 write 0xccb2a thru 0xd55b6 (0x8a8d bytes) 2024-09-04T21:55:19.953 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:19.953 INFO:teuthology.orchestra.run.smithi112.stdout:5407 read 0xd616d2f thru 0xd61abcd (0x3e9f bytes) 2024-09-04T21:55:19.953 INFO:teuthology.orchestra.run.smithi112.stdout:5408 read 0xe5a29b2 thru 0xe5abab3 (0x9102 bytes) 2024-09-04T21:55:19.970 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:19.971 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:19.972 INFO:teuthology.orchestra.run.smithi112.stdout:5409 trunc from 0xee298e1 to 0x2f8ef0b 2024-09-04T21:55:19.987 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:20.010 INFO:teuthology.orchestra.run.smithi112.stdout:5410 write 0x3329b64 thru 0x3338294 (0xe731 bytes) 2024-09-04T21:55:20.021 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:20.021 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:20.029 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:20.028+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:20.037 INFO:teuthology.orchestra.run.smithi112.stdout:5413 write 0x68e270e thru 0x68e9668 (0x6f5b bytes) 2024-09-04T21:55:20.048 INFO:teuthology.orchestra.run.smithi112.stdout:5415 punch from 0x17b733 to 0x180e0d, (0x56da bytes) 2024-09-04T21:55:20.068 INFO:teuthology.orchestra.run.smithi112.stdout:5416 writesame 0xb084d0c thru 0xb092f48 data_size 0x1(0xe23d bytes) 2024-09-04T21:55:20.068 INFO:teuthology.orchestra.run.smithi112.stdout:5417 write 0x144a5cf thru 0x144c9ed (0x241f bytes) 2024-09-04T21:55:20.074 INFO:teuthology.orchestra.run.smithi112.stdout:5418 write 0xa35213b thru 0xa356e60 (0x4d26 bytes) 2024-09-04T21:55:20.082 INFO:teuthology.orchestra.run.smithi112.stdout:5419 writesame 0xab81f00 thru 0xab8c67b data_size 0x36(0xa77c bytes) 2024-09-04T21:55:20.097 INFO:teuthology.orchestra.run.smithi112.stdout:5421 read 0x26cb360 thru 0x26cca10 (0x16b1 bytes) 2024-09-04T21:55:20.098 INFO:teuthology.orchestra.run.smithi112.stdout:5422 writesame 0x969bb39 thru 0x96a4e18 data_size 0x32(0x92e0 bytes) 2024-09-04T21:55:20.123 INFO:teuthology.orchestra.run.smithi112.stdout:5423 trunc from 0xb092f49 to 0x53fac51 2024-09-04T21:55:20.130 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:20.128+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:20.142 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:20.142 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:20.149 INFO:teuthology.orchestra.run.smithi112.stdout:5424 writesame 0x12c3644 thru 0x12cfad5 data_size 0x3(0xc492 bytes) 2024-09-04T21:55:20.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:20.208 INFO:teuthology.orchestra.run.smithi112.stdout:5426 write 0xd3b750e thru 0xd3bf3d3 (0x7ec6 bytes) 2024-09-04T21:55:20.219 INFO:teuthology.orchestra.run.smithi112.stdout:5429 writesame 0x45b5611 thru 0x45ba675 data_size 0x1(0x5065 bytes) 2024-09-04T21:55:20.231 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:20.228+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:20.236 INFO:teuthology.orchestra.run.smithi112.stdout:5431 write 0xd342037 thru 0xd34e555 (0xc51f bytes) 2024-09-04T21:55:20.250 INFO:teuthology.orchestra.run.smithi112.stdout:5434 write 0x310eddc thru 0x3115dbc (0x6fe1 bytes) 2024-09-04T21:55:20.261 INFO:teuthology.orchestra.run.smithi112.stdout:5435 read 0x8f8b7b4 thru 0x8f8dd45 (0x2592 bytes) 2024-09-04T21:55:20.261 INFO:teuthology.orchestra.run.smithi112.stdout:5436 read 0x6d5d0ec thru 0x6d5f785 (0x269a bytes) 2024-09-04T21:55:20.261 INFO:teuthology.orchestra.run.smithi112.stdout:5437 punch from 0x671a7aa to 0x671e531, (0x3d87 bytes) 2024-09-04T21:55:20.261 INFO:teuthology.orchestra.run.smithi112.stdout:5438 write 0x8d48155 thru 0x8d52c47 (0xaaf3 bytes) 2024-09-04T21:55:20.282 INFO:teuthology.orchestra.run.smithi112.stdout:5439 trunc from 0xd3bf3d4 to 0x77379dc 2024-09-04T21:55:20.308 INFO:teuthology.orchestra.run.smithi112.stdout:5442 read 0x18756a6 thru 0x1875bde (0x539 bytes) 2024-09-04T21:55:20.308 INFO:teuthology.orchestra.run.smithi112.stdout:5444 punch from 0x51e554a to 0x51f2053, (0xcb09 bytes) 2024-09-04T21:55:20.308 INFO:teuthology.orchestra.run.smithi112.stdout:5447 write 0x58384a9 thru 0x584577a (0xd2d2 bytes) 2024-09-04T21:55:20.321 INFO:teuthology.orchestra.run.smithi112.stdout:5448 punch from 0x635f229 to 0x635fb51, (0x928 bytes) 2024-09-04T21:55:20.332 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:20.328+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:20.332 INFO:teuthology.orchestra.run.smithi112.stdout:5449 trunc from 0x77379dc to 0xc4d4f1d 2024-09-04T21:55:20.340 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:20.340 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:20.345 INFO:teuthology.orchestra.run.smithi112.stdout:5450 write 0x540e5df thru 0x541a089 (0xbaab bytes) 2024-09-04T21:55:20.356 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:20.358 INFO:teuthology.orchestra.run.smithi112.stdout:5453 write 0x4bd554f thru 0x4bde797 (0x9249 bytes) 2024-09-04T21:55:20.383 INFO:teuthology.orchestra.run.smithi112.stdout:5454 write 0xd52b40d thru 0xd52ceee (0x1ae2 bytes) 2024-09-04T21:55:20.407 INFO:teuthology.orchestra.run.smithi112.stdout:5455 trunc from 0xd52ceef to 0x2675720 2024-09-04T21:55:20.433 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:20.428+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:20.442 INFO:teuthology.orchestra.run.smithi112.stdout:5456 trunc from 0x2675720 to 0x2cd0c6 2024-09-04T21:55:20.470 INFO:teuthology.orchestra.run.smithi112.stdout:5457 punch from 0xd2b6c to 0xd54e6, (0x297a bytes) 2024-09-04T21:55:20.473 INFO:teuthology.orchestra.run.smithi112.stdout:5458 trunc from 0x2cd0c6 to 0x11f7700 2024-09-04T21:55:20.489 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:20.489 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:20.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:20.517 INFO:teuthology.orchestra.run.smithi112.stdout:5461 write 0x9f29610 thru 0x9f336a8 (0xa099 bytes) 2024-09-04T21:55:20.533 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:20.528+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:5465 writesame 0xc40b132 thru 0xc4186e4 data_size 0x1(0xd5b3 bytes) 2024-09-04T21:55:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:5469 punch from 0x432df55 to 0x433009b, (0x2146 bytes) 2024-09-04T21:55:20.543 INFO:teuthology.orchestra.run.smithi112.stdout:5472 write 0x57cafc thru 0x57fdd2 (0x32d7 bytes) 2024-09-04T21:55:20.544 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:20.557 INFO:teuthology.orchestra.run.smithi112.stdout:5474 trunc from 0xc4186e5 to 0x4c95af4 2024-09-04T21:55:20.580 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:20.596 INFO:teuthology.orchestra.run.smithi112.stdout:5476 write 0x594147f thru 0x594a432 (0x8fb4 bytes) 2024-09-04T21:55:20.607 INFO:teuthology.orchestra.run.smithi112.stdout:5477 write 0x6b6e41 thru 0x6be11e (0x72de bytes) 2024-09-04T21:55:20.624 INFO:teuthology.orchestra.run.smithi112.stdout:5480 trunc from 0x594a433 to 0xacce66d 2024-09-04T21:55:20.634 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:20.628+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:20.649 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:20.649 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:20.653 INFO:teuthology.orchestra.run.smithi112.stdout:5481 write 0xc536171 thru 0xc5417c5 (0xb655 bytes) 2024-09-04T21:55:20.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:20.665 INFO:teuthology.orchestra.run.smithi112.stdout:5482 writesame 0x1d95d9d thru 0x1d9b5b6 data_size 0x3f(0x581a bytes) 2024-09-04T21:55:20.676 INFO:teuthology.orchestra.run.smithi112.stdout:5484 read 0x96a3b00 thru 0x96af7c2 (0xbcc3 bytes) 2024-09-04T21:55:20.687 INFO:teuthology.orchestra.run.smithi112.stdout:5485 writesame 0xcfc824b thru 0xcfca53a data_size 0x1a(0x22f0 bytes) 2024-09-04T21:55:20.687 INFO:teuthology.orchestra.run.smithi112.stdout:5486 write 0x810b88e thru 0x811282d (0x6fa0 bytes) 2024-09-04T21:55:20.713 INFO:teuthology.orchestra.run.smithi112.stdout:5487 writesame 0xeb3f95c thru 0xeb4caab data_size 0x88(0xd150 bytes) 2024-09-04T21:55:20.713 INFO:teuthology.orchestra.run.smithi112.stdout:5488 write 0x6126c5b thru 0x61298fd (0x2ca3 bytes) 2024-09-04T21:55:20.720 INFO:teuthology.orchestra.run.smithi112.stdout:5489 write 0x35a9370 thru 0x35b469a (0xb32b bytes) 2024-09-04T21:55:20.734 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:20.728+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:20.736 INFO:teuthology.orchestra.run.smithi112.stdout:5490 punch from 0x7f7c48f to 0x7f84e16, (0x8987 bytes) 2024-09-04T21:55:20.736 INFO:teuthology.orchestra.run.smithi112.stdout:5492 read 0x9041f5d thru 0x9051a59 (0xfafd bytes) 2024-09-04T21:55:20.736 INFO:teuthology.orchestra.run.smithi112.stdout:5494 read 0xadf041a thru 0xae002cf (0xfeb6 bytes) 2024-09-04T21:55:20.736 INFO:teuthology.orchestra.run.smithi112.stdout:5497 write 0x9e0bff5 thru 0x9e0f288 (0x3294 bytes) 2024-09-04T21:55:20.747 INFO:teuthology.orchestra.run.smithi112.stdout:5500 write 0xcc7ff2e thru 0xcc8a27d (0xa350 bytes) 2024-09-04T21:55:20.764 INFO:teuthology.orchestra.run.smithi112.stdout:5501 read 0x76dfc6 thru 0x7740e0 (0x611b bytes) 2024-09-04T21:55:20.765 INFO:teuthology.orchestra.run.smithi112.stdout:5502 punch from 0x8bc52f7 to 0x8bd3ab6, (0xe7bf bytes) 2024-09-04T21:55:20.765 INFO:teuthology.orchestra.run.smithi112.stdout:5504 read 0xaf92ca4 thru 0xafa068a (0xd9e7 bytes) 2024-09-04T21:55:20.765 INFO:teuthology.orchestra.run.smithi112.stdout:5505 punch from 0x47ef927 to 0x47f9c2e, (0xa307 bytes) 2024-09-04T21:55:20.765 INFO:teuthology.orchestra.run.smithi112.stdout:5506 writesame 0xa2a3464 thru 0xa2ac7a1 data_size 0x2f(0x933e bytes) 2024-09-04T21:55:20.779 INFO:teuthology.orchestra.run.smithi112.stdout:5507 write 0xcd85e9d thru 0xcd88860 (0x29c4 bytes) 2024-09-04T21:55:20.789 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:20.789 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:20.802 INFO:teuthology.orchestra.run.smithi112.stdout:5508 trunc from 0xeb4caac to 0x106bd13 2024-09-04T21:55:20.811 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:20.835 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:20.832+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:20.837 INFO:teuthology.orchestra.run.smithi112.stdout:5511 read 0x8bb5af thru 0x8bfc08 (0x465a bytes) 2024-09-04T21:55:20.838 INFO:teuthology.orchestra.run.smithi112.stdout:5512 read 0xb13f7f thru 0xb1b1a3 (0x7225 bytes) 2024-09-04T21:55:20.856 INFO:teuthology.orchestra.run.smithi112.stdout:5513 write 0x502a01a thru 0x502ef1e (0x4f05 bytes) 2024-09-04T21:55:20.899 INFO:teuthology.orchestra.run.smithi112.stdout:5514 write 0xe3e551a thru 0xe3f5462 (0xff49 bytes) 2024-09-04T21:55:20.913 INFO:teuthology.orchestra.run.smithi112.stdout:5515 write 0x34761d1 thru 0x34818f0 (0xb720 bytes) 2024-09-04T21:55:20.926 INFO:teuthology.orchestra.run.smithi112.stdout:5517 punch from 0x486b42a to 0x486d913, (0x24e9 bytes) 2024-09-04T21:55:20.927 INFO:teuthology.orchestra.run.smithi112.stdout:5520 write 0x6c6a2d4 thru 0x6c79a17 (0xf744 bytes) 2024-09-04T21:55:20.936 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:20.932+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:20.942 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:20.942 INFO:teuthology.orchestra.run.smithi112.stdout:5523 read 0x80357e thru 0x8057bc (0x223f bytes) 2024-09-04T21:55:20.943 INFO:teuthology.orchestra.run.smithi112.stdout:5524 write 0x5156d5e thru 0x5164bf1 (0xde94 bytes) 2024-09-04T21:55:20.970 INFO:teuthology.orchestra.run.smithi112.stdout:5526 trunc from 0xe3f5463 to 0x4deaf76 2024-09-04T21:55:20.982 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:20.983 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:20.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:20.999 INFO:teuthology.orchestra.run.smithi112.stdout:5527 trunc from 0x4deaf76 to 0x4d1a5ba 2024-09-04T21:55:21.011 INFO:teuthology.orchestra.run.smithi112.stdout:5528 read 0x34ca804 thru 0x34cfa10 (0x520d bytes) 2024-09-04T21:55:21.012 INFO:teuthology.orchestra.run.smithi112.stdout:5529 read 0x1990fd0 thru 0x199c70e (0xb73f bytes) 2024-09-04T21:55:21.029 INFO:teuthology.orchestra.run.smithi112.stdout:5530 write 0x96d9d80 thru 0x96dc0e4 (0x2365 bytes) 2024-09-04T21:55:21.037 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:21.032+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:21.040 INFO:teuthology.orchestra.run.smithi112.stdout:5531 trunc from 0x96dc0e5 to 0x62386cb 2024-09-04T21:55:21.058 INFO:teuthology.orchestra.run.smithi112.stdout:5533 punch from 0x104eea8 to 0x105e0f0, (0xf248 bytes) 2024-09-04T21:55:21.058 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:21.058 INFO:teuthology.orchestra.run.smithi112.stdout:5535 read 0x6583dc thru 0x65ff8a (0x7baf bytes) 2024-09-04T21:55:21.058 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:21.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:21.080 INFO:teuthology.orchestra.run.smithi112.stdout:5537 write 0xaffaee0 thru 0xb00739d (0xc4be bytes) 2024-09-04T21:55:21.091 INFO:teuthology.orchestra.run.smithi112.stdout:5539 read 0x767e0e7 thru 0x767e887 (0x7a1 bytes) 2024-09-04T21:55:21.091 INFO:teuthology.orchestra.run.smithi112.stdout:5540 write 0x7bda775 thru 0x7bea5a5 (0xfe31 bytes) 2024-09-04T21:55:21.106 INFO:teuthology.orchestra.run.smithi112.stdout:5541 trunc from 0xb00739e to 0xbfd8143 2024-09-04T21:55:21.116 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:21.116 INFO:teuthology.orchestra.run.smithi112.stdout:5543 read 0x3069100 thru 0x30706d4 (0x75d5 bytes) 2024-09-04T21:55:21.116 INFO:teuthology.orchestra.run.smithi112.stdout:5544 read 0xa879fd2 thru 0xa87ed13 (0x4d42 bytes) 2024-09-04T21:55:21.116 INFO:teuthology.orchestra.run.smithi112.stdout:5545 writesame 0x7ba36f0 thru 0x7ba3dc7 data_size 0xc(0x6d8 bytes) 2024-09-04T21:55:21.127 INFO:teuthology.orchestra.run.smithi112.stdout:5546 trunc from 0xbfd8143 to 0xeba6e36 2024-09-04T21:55:21.137 INFO:teuthology.orchestra.run.smithi112.stdout:5547 writesame 0x3684b17 thru 0x3689746 data_size 0x35(0x4c30 bytes) 2024-09-04T21:55:21.137 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:21.136+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:21.140 INFO:teuthology.orchestra.run.smithi112.stdout:5549 read 0x7fb4ed9 thru 0x7fc1d7b (0xcea3 bytes) 2024-09-04T21:55:21.141 INFO:teuthology.orchestra.run.smithi112.stdout:5551 writesame 0xa505e12 thru 0xa511c75 data_size 0xa(0xbe64 bytes) 2024-09-04T21:55:21.146 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:21.146 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:21.159 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:21.159 INFO:teuthology.orchestra.run.smithi112.stdout:5554 read 0x25b6299 thru 0x25b678b (0x4f3 bytes) 2024-09-04T21:55:21.159 INFO:teuthology.orchestra.run.smithi112.stdout:5556 read 0xc7ea797 thru 0xc7f17f2 (0x705c bytes) 2024-09-04T21:55:21.162 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5559 writesame 0xece7de4 thru 0xecef716 data_size 0x1(0x7933 bytes) 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5560 read 0x23b63df thru 0x23c058e (0xa1b0 bytes) 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5561 read 0xb012d4b thru 0xb01efdd (0xc293 bytes) 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5563 punch from 0xa633c19 to 0xa643712, (0xfaf9 bytes) 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5566 read 0xa7c0ef7 thru 0xa7c8c47 (0x7d51 bytes) 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5567 read 0x37eec4f thru 0x37f707a (0x842c bytes) 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:21.173 INFO:teuthology.orchestra.run.smithi112.stdout:5569 read 0x302322b thru 0x30298b1 (0x6687 bytes) 2024-09-04T21:55:21.174 INFO:teuthology.orchestra.run.smithi112.stdout:5570 punch from 0x319e7fa to 0x31a35dc, (0x4de2 bytes) 2024-09-04T21:55:21.174 INFO:teuthology.orchestra.run.smithi112.stdout:5571 clone 31 order 24 su 65536 sc 12 2024-09-04T21:55:21.225 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:21.225 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:21.239 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:21.236+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:21.242 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:21.243 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:21.259 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:21.260 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:21.277 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:21.278 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:21.278 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:21.282 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:21.282 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:21.296 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:21.298 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:21.298 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:21.316 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:21.316 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:21.335 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:21.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:21.340 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:21.336+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.979536531000001, 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:55:21.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:55:21.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:21.430 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.794995475999997, 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:55:21.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:55:21.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:55:21.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43351 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461700047000001, 2024-09-04T21:55:21.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051338846, 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43351 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4499999999999997e-07 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959605+0000", 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959613+0000", 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8250000000000005e-06 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959618+0000", 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.417e-06 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010746+0000", 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051127924999999998 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010835+0000", 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9444000000000006e-05 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010941+0000", 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105312 2024-09-04T21:55:21.440 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43353 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1269760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461675995, 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051544344999999998, 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.441 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43353 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959629+0000", 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959635+0000", 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0430000000000004e-06 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959640+0000", 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0320000000000003e-06 2024-09-04T21:55:21.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010976+0000", 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051336248000000001 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011067+0000", 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0826000000000001e-05 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011170+0000", 2024-09-04T21:55:21.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103507 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43355 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461653853000001, 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051760575000000003, 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43355 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7800000000000002e-07 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959651+0000", 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-04T21:55:21.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959658+0000", 2024-09-04T21:55:21.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5730000000000002e-06 2024-09-04T21:55:21.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959663+0000", 2024-09-04T21:55:21.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:21.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011205+0000", 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051541489000000003 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011296+0000", 2024-09-04T21:55:21.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1105000000000003e-05 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011409+0000", 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112927 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.452 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43358 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461630888000002, 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051974999000000001, 2024-09-04T21:55:21.453 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43358 2024-09-04T21:55:21.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:21.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9999999999999998e-07 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959674+0000", 2024-09-04T21:55:21.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959682+0000", 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9500000000000001e-06 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959687+0000", 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1420000000000001e-06 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011451+0000", 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051763862000000001 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011538+0000", 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7322000000000002e-05 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011646+0000", 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010772000000000001 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43361 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461607471000001, 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052194391, 2024-09-04T21:55:21.460 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43361 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:21.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959695+0000", 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:21.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959697+0000", 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.418e-06 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959703+0000", 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9989999999999997e-06 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959708+0000", 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.904e-06 2024-09-04T21:55:21.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011687+0000", 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051978506000000001 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011781+0000", 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4301000000000004e-05 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011889+0000", 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107796 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43362 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:21.465 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461586680000003, 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052922181999999998, 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43362 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959716+0000", 2024-09-04T21:55:21.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959718+0000", 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959725+0000", 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4660000000000002e-06 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959730+0000", 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1019999999999996e-06 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011924+0000", 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052193145000000003 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012015+0000", 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0969000000000003e-05 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012637+0000", 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062277700000000003 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43364 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1351680~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461564277000001, 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052915188000000002, 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.470 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43364 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3799999999999997e-07 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959740+0000", 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959747+0000", 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9149999999999999e-06 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959754+0000", 2024-09-04T21:55:21.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8000000000000001e-06 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012425+0000", 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052670531 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012525+0000", 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100581 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012653+0000", 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012751700000000001 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43366 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.461540628999998, 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052927226000000001, 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.475 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43366 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959762+0000", 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0399999999999996e-07 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959764+0000", 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3949999999999999e-06 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960370+0000", 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060537999999999998 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960389+0000", 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9326000000000001e-05 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012680+0000", 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052291232 2024-09-04T21:55:21.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012688+0000", 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3890000000000005e-06 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43368 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.479 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.460881944, 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052287419000000002, 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43368 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960422+0000", 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.84e-06 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960431+0000", 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960971+0000", 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000539965 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960991+0000", 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0832999999999999e-05 2024-09-04T21:55:21.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012703+0000", 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051711615000000002 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012707+0000", 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3499999999999999e-06 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43369 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.460281238, 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051701824, 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43369 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961022+0000", 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6139999999999999e-06 2024-09-04T21:55:21.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961030+0000", 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1489999999999994e-06 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961051+0000", 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961065+0000", 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4163e-05 2024-09-04T21:55:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012718+0000", 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051653533000000001 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012722+0000", 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.487 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43371 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.460216176000003, 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051652328999999997, 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43371 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961087+0000", 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:55:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961106+0000", 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9097e-05 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961146+0000", 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0281999999999998e-05 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961162+0000", 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5993000000000001e-05 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012734+0000", 2024-09-04T21:55:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051571744000000003 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012738+0000", 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43373 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.460118469000001, 2024-09-04T21:55:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051570802999999998, 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43373 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961185+0000", 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.423e-06 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961191+0000", 2024-09-04T21:55:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9429999999999997e-06 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961210+0000", 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8943000000000001e-05 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961223+0000", 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3506000000000001e-05 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012750+0000", 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051527008999999999 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012754+0000", 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9790000000000004e-06 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43375 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.460059293999997, 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051526660000000002, 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43375 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961244+0000", 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961249+0000", 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961270+0000", 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0925999999999999e-05 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961283+0000", 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2986000000000001e-05 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012765+0000", 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0514825 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012769+0000", 2024-09-04T21:55:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8079999999999998e-06 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43377 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.460002058999997, 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051486632999999997, 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43377 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961301+0000", 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961306+0000", 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.69e-06 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961323+0000", 2024-09-04T21:55:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6976000000000001e-05 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961336+0000", 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2913e-05 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012781+0000", 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051444996999999999 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012786+0000", 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9340000000000003e-06 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43379 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.459949412999997, 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051454868000000001, 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43379 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961354+0000", 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0729999999999999e-06 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961358+0000", 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6099999999999999e-06 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961375+0000", 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7091999999999998e-05 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961390+0000", 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4803e-05 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012803+0000", 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051413343 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012807+0000", 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9469999999999996e-06 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43381 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.459892985000003, 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051415832000000002, 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43381 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961410+0000", 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961415+0000", 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.977e-06 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961431+0000", 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6016e-05 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961444+0000", 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2816000000000001e-05 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012821+0000", 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051376809000000002 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012825+0000", 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0910000000000003e-06 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43383 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.459839856999999, 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051378030999999998, 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43383 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961463+0000", 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961473+0000", 2024-09-04T21:55:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961535+0000", 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2787999999999996e-05 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961555+0000", 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9646000000000001e-05 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012836+0000", 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051281032999999997 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012840+0000", 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9840000000000003e-06 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43385 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.459711910000003, 2024-09-04T21:55:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051265590999999999, 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43385 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961591+0000", 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961597+0000", 2024-09-04T21:55:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961615+0000", 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7836e-05 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961628+0000", 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2976000000000001e-05 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012851+0000", 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051223109000000003 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012856+0000", 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7870000000000001e-06 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.522 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:21.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:21.525 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:21.520+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:21.599 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:55:21.625 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:21.620+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:21.656 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:21.657 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:21.672 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:21.726 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:21.724+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:21.827 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:21.824+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:21.833 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:21.833 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:21.848 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:21.928 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:21.924+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:55:21.982 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 40.617988758999999, 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:55:21.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49175 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 475136~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.185292334000003, 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021719833000000001, 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49175 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793193+0000", 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4e-07 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793196+0000", 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-04T21:55:21.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793206+0000", 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4229999999999994e-06 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793213+0000", 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4370000000000001e-06 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814388+0000", 2024-09-04T21:55:21.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021174454999999998 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814604+0000", 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021587900000000001 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814912+0000", 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030848600000000001 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49290 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 40960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.185261670999999, 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022433502000000001, 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49290 2024-09-04T21:55:21.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793224+0000", 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6599999999999996e-07 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793227+0000", 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793240+0000", 2024-09-04T21:55:21.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-05 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793252+0000", 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2843000000000001e-05 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815053+0000", 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021800657000000001 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815301+0000", 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024764399999999999 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815656+0000", 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035569200000000001 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:21.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49297 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 106496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.185220979, 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022960876000000002, 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49297 2024-09-04T21:55:21.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793268+0000", 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4460000000000001e-06 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793277+0000", 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2299999999999997e-06 2024-09-04T21:55:22.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793284+0000", 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9530000000000002e-06 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815785+0000", 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022500857999999999 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815996+0000", 2024-09-04T21:55:22.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211065 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816224+0000", 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022851699999999999 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49303 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 172032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:22.003 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.185191156999998, 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023578159000000001, 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49303 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793294+0000", 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793297+0000", 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9500000000000001e-06 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793306+0000", 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2140000000000002e-06 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793313+0000", 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816342+0000", 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023028835000000001 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816580+0000", 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023813 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816872+0000", 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029135000000000001 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49309 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 237568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.185120195000003, 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024080300999999998, 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.008 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49309 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793365+0000", 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.054e-06 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793374+0000", 2024-09-04T21:55:22.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793388+0000", 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3403e-05 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793396+0000", 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8830000000000006e-06 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816958+0000", 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023561992 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817170+0000", 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021269799999999999 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817445+0000", 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027435799999999999 2024-09-04T21:55:22.012 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49326 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 303104~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.185045273, 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024582155000000001, 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.013 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49326 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793440+0000", 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.003e-06 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793445+0000", 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6840000000000004e-06 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793464+0000", 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9085000000000001e-05 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793478+0000", 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3716999999999999e-05 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817523+0000", 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024045704000000001 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817738+0000", 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021439899999999999 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818021+0000", 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028356300000000002 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49332 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.018 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.184993153999997, 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025083616, 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49332 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793492+0000", 2024-09-04T21:55:22.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793496+0000", 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793506+0000", 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9729999999999992e-06 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793516+0000", 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6269999999999995e-06 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818099+0000", 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024583880999999998 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818307+0000", 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020786200000000001 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818575+0000", 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026768400000000002 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49338 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.023 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.184958967999997, 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025617277000000001, 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49338 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5600000000000005e-07 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793533+0000", 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1930000000000004e-06 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793542+0000", 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0669999999999998e-06 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793552+0000", 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1290000000000002e-06 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818661+0000", 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025109056000000001 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818870+0000", 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000208929 2024-09-04T21:55:22.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819143+0000", 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027324700000000001 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49344 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.184922983, 2024-09-04T21:55:22.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026178261000000001, 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49344 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:22.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793565+0000", 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8930000000000001e-06 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793574+0000", 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793581+0000", 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8399999999999997e-06 2024-09-04T21:55:22.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819234+0000", 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025653441999999999 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819463+0000", 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022879500000000001 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819740+0000", 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027665399999999998 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51043 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851385811999997, 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022470637000000002, 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.033 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51043 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127100+0000", 2024-09-04T21:55:22.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127114+0000", 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4017e-05 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127128+0000", 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4005000000000001e-05 2024-09-04T21:55:22.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127139+0000", 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1007e-05 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148542+0000", 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021403575000000001 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148727+0000", 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018442900000000001 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149569+0000", 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084262400000000004 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51044 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 262144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:22.038 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851331205999998, 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023138729, 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51044 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127154+0000", 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127158+0000", 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.172e-06 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127194+0000", 2024-09-04T21:55:22.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5222000000000003e-05 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127206+0000", 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2159e-05 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149626+0000", 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022420476000000002 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149779+0000", 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015296299999999999 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150292+0000", 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051277400000000002 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51045 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 360448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.043 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851260658000001, 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023446379, 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51045 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127225+0000", 2024-09-04T21:55:22.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127230+0000", 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4110000000000004e-06 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127245+0000", 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4714e-05 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127256+0000", 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0787e-05 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150350+0000", 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023094001999999999 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150502+0000", 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151805 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150670+0000", 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016861 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51046 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.048 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851212330999999, 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023742831999999998, 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51046 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127273+0000", 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.23e-07 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127277+0000", 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9650000000000002e-06 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127291+0000", 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3549e-05 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127302+0000", 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1270000000000001e-05 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150727+0000", 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023424996999999999 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150882+0000", 2024-09-04T21:55:22.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015530200000000001 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151015+0000", 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013292600000000001 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51047 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 557056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:22.053 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851169196999997, 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024031869000000001, 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51047 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127316+0000", 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5300000000000003e-07 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127320+0000", 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0749999999999999e-06 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127334+0000", 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3587000000000001e-05 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127344+0000", 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0461e-05 2024-09-04T21:55:22.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151070+0000", 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023725311999999998 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151219+0000", 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014918599999999999 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151347+0000", 2024-09-04T21:55:22.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012839500000000001 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51048 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851126720000003, 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024346703000000001, 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.058 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51048 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:22.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127359+0000", 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.85e-07 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127375+0000", 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6084000000000001e-05 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127389+0000", 2024-09-04T21:55:22.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.412e-05 2024-09-04T21:55:22.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127400+0000", 2024-09-04T21:55:22.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-04T21:55:22.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151402+0000", 2024-09-04T21:55:22.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024002787000000001 2024-09-04T21:55:22.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151573+0000", 2024-09-04T21:55:22.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170941 2024-09-04T21:55:22.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151705+0000", 2024-09-04T21:55:22.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131161 2024-09-04T21:55:22.064 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.064 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51049 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851057273999999, 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024613729000000001, 2024-09-04T21:55:22.065 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51049 2024-09-04T21:55:22.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.067 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:22.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:22.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127428+0000", 2024-09-04T21:55:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6700000000000002e-07 2024-09-04T21:55:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127433+0000", 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0000000000000004e-06 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127447+0000", 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4224e-05 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127459+0000", 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1112e-05 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151761+0000", 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024302397999999999 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151914+0000", 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015300399999999999 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152041+0000", 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127024 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51050 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.851010971000001, 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024957087999999999, 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51050 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127479+0000", 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7160000000000002e-06 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127494+0000", 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4997000000000001e-05 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127506+0000", 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.168e-05 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152104+0000", 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024597810000000001 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152273+0000", 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016888100000000001 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152431+0000", 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000158097 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51051 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.850963477000001, 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025255452000000001, 2024-09-04T21:55:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51051 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127522+0000", 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127527+0000", 2024-09-04T21:55:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6299999999999997e-06 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127541+0000", 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4225e-05 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127552+0000", 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0971999999999999e-05 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152484+0000", 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024931803999999998 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152641+0000", 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015691699999999999 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152776+0000", 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013595400000000001 2024-09-04T21:55:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:106971 3.0 e82/38 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head v 82'257, mlcod=81'254)", 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.348669148999999, 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045768883000000003, 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629817+0000", 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9400000000000001e-06 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629834+0000", 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6246999999999999e-05 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629860+0000", 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6628999999999999e-05 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629873+0000", 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3216e-05 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629999+0000", 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012590799999999999 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.630064+0000", 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4546000000000002e-05 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675498+0000", 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045433922000000002 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675584+0000", 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6475000000000001e-05 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.087 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:22.088 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-09-04T21:55:22.089 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:22.089 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:22.084+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:22.190 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:22.184+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:22.215 INFO:teuthology.orchestra.run.smithi112.stdout:5572 write 0x8df9112 thru 0x8e04fb2 (0xbea1 bytes) 2024-09-04T21:55:22.215 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:22.215 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:22.216 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:22.216 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:22.232 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:22.259 INFO:teuthology.orchestra.run.smithi112.stdout:5574 read 0x8dd7c10 thru 0x8ddf567 (0x7958 bytes) 2024-09-04T21:55:22.259 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:22.259 INFO:teuthology.orchestra.run.smithi112.stdout:5577 read 0xb17f21c thru 0xb18af1d (0xbd02 bytes) 2024-09-04T21:55:22.259 INFO:teuthology.orchestra.run.smithi112.stdout:5579 read 0xa73f392 thru 0xa740299 (0xf08 bytes) 2024-09-04T21:55:22.260 INFO:teuthology.orchestra.run.smithi112.stdout:5581 read 0x59aa49a thru 0x59b1d2b (0x7892 bytes) 2024-09-04T21:55:22.260 INFO:teuthology.orchestra.run.smithi112.stdout:5582 writesame 0x5b4b280 thru 0x5b54382 data_size 0x1(0x9103 bytes) 2024-09-04T21:55:22.291 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:22.288+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:22.307 INFO:teuthology.orchestra.run.smithi112.stdout:5583 write 0xc3883bc thru 0xc394c8b (0xc8d0 bytes) 2024-09-04T21:55:22.313 INFO:teuthology.orchestra.run.smithi112.stdout:5584 writesame 0xa509bfd thru 0xa51236f data_size 0x19(0x8773 bytes) 2024-09-04T21:55:22.352 INFO:teuthology.orchestra.run.smithi112.stdout:5585 write 0xaaa7ed1 thru 0xaab57d8 (0xd908 bytes) 2024-09-04T21:55:22.376 INFO:teuthology.orchestra.run.smithi112.stdout:5586 read 0xf36a7d thru 0xf39198 (0x271c bytes) 2024-09-04T21:55:22.377 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:22.377 INFO:teuthology.orchestra.run.smithi112.stdout:5588 writesame 0xe067acf thru 0xe06db65 data_size 0x1(0x6097 bytes) 2024-09-04T21:55:22.391 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:22.388+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:22.392 INFO:teuthology.orchestra.run.smithi112.stdout:5590 trunc from 0xecef717 to 0xe541491 2024-09-04T21:55:22.411 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:22.412 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:22.412 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:22.412 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:22.412 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:22.412 INFO:teuthology.orchestra.run.smithi112.stdout:5592 punch from 0x27d0a9d to 0x27d9221, (0x8784 bytes) 2024-09-04T21:55:22.412 INFO:teuthology.orchestra.run.smithi112.stdout:5593 writesame 0x9df7302 thru 0x9dfbabd data_size 0x2(0x47bc bytes) 2024-09-04T21:55:22.419 INFO:teuthology.orchestra.run.smithi112.stdout:5594 write 0x45d7994 thru 0x45dba7f (0x40ec bytes) 2024-09-04T21:55:22.422 INFO:teuthology.orchestra.run.smithi112.stdout:5596 read 0x248b063 thru 0x2497189 (0xc127 bytes) 2024-09-04T21:55:22.423 INFO:teuthology.orchestra.run.smithi112.stdout:5598 read 0x6dac672 thru 0x6db77fa (0xb189 bytes) 2024-09-04T21:55:22.424 INFO:teuthology.orchestra.run.smithi112.stdout:5599 write 0x8a4cddf thru 0x8a557c1 (0x89e3 bytes) 2024-09-04T21:55:22.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:22.444 INFO:teuthology.orchestra.run.smithi112.stdout:5600 read 0x5b32295 thru 0x5b33d6a (0x1ad6 bytes) 2024-09-04T21:55:22.445 INFO:teuthology.orchestra.run.smithi112.stdout:5601 read 0xc03378c thru 0xc03f9f2 (0xc267 bytes) 2024-09-04T21:55:22.445 INFO:teuthology.orchestra.run.smithi112.stdout:5602 read 0x6d0e928 thru 0x6d177d4 (0x8ead bytes) 2024-09-04T21:55:22.463 INFO:teuthology.orchestra.run.smithi112.stdout:5603 trunc from 0xe541491 to 0x62a99a3 2024-09-04T21:55:22.493 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:22.488+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:22.513 INFO:teuthology.orchestra.run.smithi112.stdout:5604 read 0x2a8bd7b thru 0x2a92232 (0x64b8 bytes) 2024-09-04T21:55:22.513 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:22.536 INFO:teuthology.orchestra.run.smithi112.stdout:5610 write 0xecc0764 thru 0xecc2c97 (0x2534 bytes) 2024-09-04T21:55:22.539 INFO:teuthology.orchestra.run.smithi112.stdout:5611 punch from 0x9b46215 to 0x9b55e31, (0xfc1c bytes) 2024-09-04T21:55:22.548 INFO:teuthology.orchestra.run.smithi112.stdout:5613 trunc from 0xecc2c98 to 0xa5d6094 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:22.568 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.950705841999998, 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:22.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.939700598999998, 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.575 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:22.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.396017825000001, 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:55:22.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:55:22.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395993849, 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:22.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.591 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395953163000001, 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:22.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.596 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395919763999999, 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:22.602 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395876493000003, 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:22.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:22.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395831811999997, 2024-09-04T21:55:22.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.609 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395770656000003, 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.615 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:22.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395740615000001, 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.620 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:22.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:22.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:22.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395695844000002, 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.626 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:22.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:22.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395647552, 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.631 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.632 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.632 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.632 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.632 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.632 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395611907000003, 2024-09-04T21:55:22.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:22.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.642 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395362691000003, 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.643 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:22.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:22.648 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.395323822000002, 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:22.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:22.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.654 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.654 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.390177076999997, 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.655 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:22.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:22.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.660 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.390133818000002, 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:22.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.390089947, 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.666 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.667 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.667 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.667 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.667 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:22.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.667 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.671 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.390053668999997, 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:22.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:22.675 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.390027562999997, 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:22.676 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:22.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:22.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:22.680 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:22.681 INFO:teuthology.orchestra.run.smithi112.stdout:5614 trunc from 0xa5d6094 to 0xed714ac 2024-09-04T21:55:22.681 INFO:teuthology.orchestra.run.smithi112.stdout:5616 clone 32 order 22 su 262144 sc 15 2024-09-04T21:55:22.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:22.683 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:55:22.683 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:22.680+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:22.783 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:22.780+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:22.854 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:22.854 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:22.854 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:22.854 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:22.868 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:22.884 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:22.880+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:22.985 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:22.980+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:22.990 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:22.990 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:23.004 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:23.093 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:23.092+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:23.172 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494575315000006, 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494548530000003, 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:23.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494519596999993, 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:23.182 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:23.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494495950000001, 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.187 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494476168999995, 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.191 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494456774, 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:23.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:23.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494437500999993, 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494418279000001, 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:23.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494399442000002, 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.209 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.213 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494380039000006, 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:23.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:23.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494358982999998, 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.218 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.222 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494326805, 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.226 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494300584000001, 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.227 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.231 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494272374999994, 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:23.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494252334999999, 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.236 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494232952999994, 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:23.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:23.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494209365000003, 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.245 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:23.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494189047999996, 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:23.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494169495999998, 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.254 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:23.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.494149737000001, 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:23.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:23.263 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:23.264 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:23.265 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:23.260+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:23.340 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:23.365 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:23.360+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:23.434 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:23.434 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:23.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:23.466 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:23.464+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:23.495 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:23.496 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:23.512 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:23.512 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:23.529 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:23.546 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:23.547 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:23.547 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:23.567 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:23.564+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:23.569 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:23.570 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:23.570 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:23.585 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:23.588 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:23.589 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:23.615 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:23.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:23.668 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:23.664+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:23.703 INFO:teuthology.orchestra.run.smithi112.stdout:leaving image image_client.0-clone31 intact 2024-09-04T21:55:23.705 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:23.705 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:23.720 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:23.768 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:23.764+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:23.869 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:23.868+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:23.891 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:23.891 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:23.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:23.955 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-09-04T21:55:23.970 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:23.968+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:24.071 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:24.068+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:24.078 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:24.078 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:24.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:24.185 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:24.184+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:24.259 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:24.259 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:24.273 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:24.274 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:24.268+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:24.351 INFO:teuthology.orchestra.run.smithi112.stdout:5617 write 0xe357b26 thru 0xe35c432 (0x490d bytes) 2024-09-04T21:55:24.372 INFO:teuthology.orchestra.run.smithi112.stdout:5619 trunc from 0xed714ac to 0xaf5d252 2024-09-04T21:55:24.374 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:24.368+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:24.453 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7] out_osds: [1, 0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-04T21:55:24.453 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:55:24.453 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-04T21:55:24.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-04T21:55:24.462 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:24.462 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:24.475 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:24.472+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:24.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:24.488 INFO:teuthology.orchestra.run.smithi112.stdout:5620 read 0x3f056ed thru 0x3f0effc (0x9910 bytes) 2024-09-04T21:55:24.489 INFO:teuthology.orchestra.run.smithi112.stdout:5622 write 0x69a912d thru 0x69b02e1 (0x71b5 bytes) 2024-09-04T21:55:24.508 INFO:teuthology.orchestra.run.smithi112.stdout:5624 trunc from 0xaf5d252 to 0x40ece1a 2024-09-04T21:55:24.576 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:24.572+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:24.651 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:24.664 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:24.666 INFO:teuthology.orchestra.run.smithi112.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:24.666 INFO:teuthology.orchestra.run.smithi112.stderr:0 2024-09-04T21:55:24.666 INFO:teuthology.orchestra.run.smithi112.stderr:1 2024-09-04T21:55:24.666 INFO:teuthology.orchestra.run.smithi112.stderr:2 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:abort 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:assert 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:bench [] [] [] [] 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs debug_inject_read_zeros 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs files list 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs stats 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:bluestore allocator dump block 2024-09-04T21:55:24.667 INFO:teuthology.orchestra.run.smithi112.stderr:Error EINVAL: invalid command 2024-09-04T21:55:24.668 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-04T21:55:24.668 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-04T21:55:24.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:24.684 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:55:24.700 INFO:teuthology.orchestra.run.smithi112.stdout:5625 read 0x630056 thru 0x63e58d (0xe538 bytes) 2024-09-04T21:55:24.702 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:24.700+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:24.721 INFO:teuthology.orchestra.run.smithi112.stdout:5626 write 0xac51470 thru 0xac5ba13 (0xa5a4 bytes) 2024-09-04T21:55:24.728 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:24.728 INFO:teuthology.orchestra.run.smithi112.stdout:5629 read 0x9130b60 thru 0x913287a (0x1d1b bytes) 2024-09-04T21:55:24.728 INFO:teuthology.orchestra.run.smithi112.stdout:5631 writesame 0x246ef33 thru 0x2473b28 data_size 0x15(0x4bf6 bytes) 2024-09-04T21:55:24.741 INFO:teuthology.orchestra.run.smithi112.stdout:5632 read 0x765a7b2 thru 0x7661a09 (0x7258 bytes) 2024-09-04T21:55:24.757 INFO:teuthology.orchestra.run.smithi112.stdout:5633 write 0xe04f711 thru 0xe051caf (0x259f bytes) 2024-09-04T21:55:24.762 INFO:teuthology.orchestra.run.smithi112.stdout:5635 read 0x11943a4 thru 0x119ae87 (0x6ae4 bytes) 2024-09-04T21:55:24.763 INFO:teuthology.orchestra.run.smithi112.stdout:5636 trunc from 0xe051cb0 to 0xcf065a7 2024-09-04T21:55:24.775 INFO:teuthology.orchestra.run.smithi112.stdout:5637 write 0x6da4cd6 thru 0x6da9480 (0x47ab bytes) 2024-09-04T21:55:24.778 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:24.776+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:24.779 INFO:teuthology.orchestra.run.smithi112.stdout:5638 write 0xb73e9cf thru 0xb74b120 (0xc752 bytes) 2024-09-04T21:55:24.786 INFO:teuthology.orchestra.run.smithi112.stdout:5639 trunc from 0xcf065a7 to 0xdf5ccf8 2024-09-04T21:55:24.797 INFO:teuthology.orchestra.run.smithi112.stdout:5640 writesame 0xb3a5976 thru 0xb3b147f data_size 0x2(0xbb0a bytes) 2024-09-04T21:55:24.821 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:24.821 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:24.823 INFO:teuthology.orchestra.run.smithi112.stdout:5642 write 0x24cbf72 thru 0x24db01c (0xf0ab bytes) 2024-09-04T21:55:24.839 INFO:teuthology.orchestra.run.smithi112.stdout:5643 write 0x90ece7a thru 0x90f7ed3 (0xb05a bytes) 2024-09-04T21:55:24.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:24.845 INFO:teuthology.orchestra.run.smithi112.stdout:5644 read 0xa993319 thru 0xa994629 (0x1311 bytes) 2024-09-04T21:55:24.853 INFO:teuthology.orchestra.run.smithi112.stdout:5645 write 0xe154d1d thru 0xe162b5d (0xde41 bytes) 2024-09-04T21:55:24.857 INFO:teuthology.orchestra.run.smithi112.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:0 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:1 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:2 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:abort 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:assert 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:bench [] [] [] [] 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs debug_inject_read_zeros 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs files list 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs stats 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:bluestore allocator dump block 2024-09-04T21:55:24.858 INFO:teuthology.orchestra.run.smithi112.stderr:Error EINVAL: invalid command 2024-09-04T21:55:24.859 INFO:teuthology.orchestra.run.smithi112.stdout:5646 read 0xaeca36c thru 0xaed6f36 (0xcbcb bytes) 2024-09-04T21:55:24.859 INFO:teuthology.orchestra.run.smithi112.stdout:5649 punch from 0x9c9dca0 to 0x9ca26d8, (0x4a38 bytes) 2024-09-04T21:55:24.859 INFO:teuthology.orchestra.run.smithi112.stdout:5651 punch from 0x2b8291c to 0x2b87a46, (0x512a bytes) 2024-09-04T21:55:24.859 INFO:teuthology.orchestra.run.smithi112.stdout:5652 writesame 0xe2ffc7 thru 0xe33de0 data_size 0x1(0x3e1a bytes) 2024-09-04T21:55:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:24.861 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-04T21:55:24.861 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-04T21:55:24.876 INFO:teuthology.orchestra.run.smithi112.stdout:5654 read 0x35e17b2 thru 0x35e8224 (0x6a73 bytes) 2024-09-04T21:55:24.876 INFO:teuthology.orchestra.run.smithi112.stdout:5655 read 0xbdee89 thru 0xbe5a38 (0x6bb0 bytes) 2024-09-04T21:55:24.877 INFO:teuthology.orchestra.run.smithi112.stdout:5657 write 0xa306266 thru 0xa311991 (0xb72c bytes) 2024-09-04T21:55:24.882 INFO:teuthology.orchestra.run.smithi112.stdout:5658 punch from 0xc39bb87 to 0xc3aa6b4, (0xeb2d bytes) 2024-09-04T21:55:24.883 INFO:teuthology.orchestra.run.smithi112.stdout:5660 trunc from 0xe162b5e to 0xeaecbcd 2024-09-04T21:55:24.887 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:24.884+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:24.891 INFO:teuthology.orchestra.run.smithi112.stdout:5661 writesame 0xb25785a thru 0xb261b0b data_size 0x77(0xa2b2 bytes) 2024-09-04T21:55:24.896 INFO:teuthology.orchestra.run.smithi112.stdout:5662 clone 33 order 24 su 4194304 sc 6 2024-09-04T21:55:24.979 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:24.979 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:24.979 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:24.976+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:24.993 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:25.007 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-04T21:55:25.007 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-04T21:55:25.093 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:25.092+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:25.160 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:25.160 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:25.177 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:25.180 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:25.176+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:0 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:1 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:2 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:abort 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:assert 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:bench [] [] [] [] 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs debug_inject_read_zeros 2024-09-04T21:55:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs files list 2024-09-04T21:55:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs stats 2024-09-04T21:55:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:bluestore allocator dump block 2024-09-04T21:55:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:Error EINVAL: invalid command 2024-09-04T21:55:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:25.203 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-04T21:55:25.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-04T21:55:25.282 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:25.276+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:25.294 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:25.294 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:25.309 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:25.383 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:25.380+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:0 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:1 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:2 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:abort 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:assert 2024-09-04T21:55:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:bench [] [] [] [] 2024-09-04T21:55:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs debug_inject_read_zeros 2024-09-04T21:55:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs files list 2024-09-04T21:55:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs stats 2024-09-04T21:55:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:bluestore allocator dump block 2024-09-04T21:55:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:Error EINVAL: invalid command 2024-09-04T21:55:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:25.402 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-04T21:55:25.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-04T21:55:25.476 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:25.476 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:25.483 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:25.480+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:25.491 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:25.584 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:25.580+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:0 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:1 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:2 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:abort 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:assert 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:bench [] [] [] [] 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs debug_inject_read_zeros 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs files list 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:bluefs stats 2024-09-04T21:55:25.589 INFO:teuthology.orchestra.run.smithi112.stderr:bluestore allocator dump block 2024-09-04T21:55:25.590 INFO:teuthology.orchestra.run.smithi112.stderr:Error EINVAL: invalid command 2024-09-04T21:55:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:25.592 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-04T21:55:25.609 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:55:25.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:25.666 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:25.666 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:25.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:25.685 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:25.680+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:25.690 INFO:teuthology.orchestra.run.smithi112.stdout:truncating image image_client.0-clone32 from 0xeaecbcd (overlap 0x40ece1a) to 0x0 2024-09-04T21:55:25.772 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:25.785 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:25.784+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:25.788 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:25.788 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:25.804 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:25.804 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:25.815 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-09-04T21:55:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:25.820 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:25.820 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:25.840 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:25.840 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:25.851 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:25.851 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:25.861 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:25.861 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:25.865 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:25.881 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:25.881 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:25.884 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:25.911 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:25.908+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:25.983 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:25.983 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:25.984 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:25.984 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:25.986 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:25.980+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:25.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:26.087 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:26.084+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:26.154 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:26.154 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:26.154 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:26.154 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:26.154 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:26.167 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:26.188 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:26.184+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:26.211 INFO:teuthology.orchestra.run.smithi112.stdout:5665 read 0xe7c3b0a thru 0xe7d1529 (0xda20 bytes) 2024-09-04T21:55:26.213 INFO:teuthology.orchestra.run.smithi112.stdout:5666 writesame 0xb29d4e3 thru 0xb2aa1f6 data_size 0x96(0xcd14 bytes) 2024-09-04T21:55:26.228 INFO:teuthology.orchestra.run.smithi112.stdout:5667 write 0x844345 thru 0x84d260 (0x8f1c bytes) 2024-09-04T21:55:26.245 INFO:teuthology.orchestra.run.smithi112.stdout:5668 read 0x65c7735 thru 0x65d1d00 (0xa5cc bytes) 2024-09-04T21:55:26.245 INFO:teuthology.orchestra.run.smithi112.stdout:5669 read 0xd496b6a thru 0xd49bf06 (0x539d bytes) 2024-09-04T21:55:26.246 INFO:teuthology.orchestra.run.smithi112.stdout:5670 write 0xc4b286d thru 0xc4c0040 (0xd7d4 bytes) 2024-09-04T21:55:26.250 INFO:teuthology.orchestra.run.smithi112.stdout:5672 write 0x56e8365 thru 0x56ebe87 (0x3b23 bytes) 2024-09-04T21:55:26.278 INFO:teuthology.orchestra.run.smithi112.stdout:5673 read 0x6ceba92 thru 0x6cf224c (0x67bb bytes) 2024-09-04T21:55:26.278 INFO:teuthology.orchestra.run.smithi112.stdout:5674 punch from 0xb730668 to 0xb738779, (0x8111 bytes) 2024-09-04T21:55:26.279 INFO:teuthology.orchestra.run.smithi112.stdout:5675 punch from 0x36c73e8 to 0x36cf443, (0x805b bytes) 2024-09-04T21:55:26.279 INFO:teuthology.orchestra.run.smithi112.stdout:5676 writesame 0xc337d9d thru 0xc345686 data_size 0x2d(0xd8ea bytes) 2024-09-04T21:55:26.283 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.284 INFO:teuthology.orchestra.run.smithi112.stdout:5678 read 0xcb4f20c thru 0xcb5b6a1 (0xc496 bytes) 2024-09-04T21:55:26.289 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:26.284+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:26.290 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.673325282, 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.662320039000001, 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118637264999997, 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:55:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118613289000002, 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118572602999997, 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118539204000001, 2024-09-04T21:55:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118495932999998, 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118451252, 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:26.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118390095999999, 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:26.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:26.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118360054999997, 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:26.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:26.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:26.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118315283999998, 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.338 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:26.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118266992000002, 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.343 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:26.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.118231346999998, 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.348 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:26.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:26.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.117982130999998, 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.353 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:26.357 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.117943261999997, 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:26.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.112796517, 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.362 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:26.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.112753257999998, 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.367 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.112709387000002, 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:26.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:26.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.112673108999999, 2024-09-04T21:55:26.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:26.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:26.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.112647002999999, 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:26.381 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:26.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:26.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:26.385 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:26.387 INFO:teuthology.orchestra.run.smithi112.stdout:5679 trunc from 0xeaecbcd to 0x6ebc38d 2024-09-04T21:55:26.387 INFO:teuthology.orchestra.run.smithi112.stdout:5682 writesame 0x56f3a6 thru 0x578d03 data_size 0x43(0x995e bytes) 2024-09-04T21:55:26.387 INFO:teuthology.orchestra.run.smithi112.stdout:5683 punch from 0x3adac15 to 0x3ae622b, (0xb616 bytes) 2024-09-04T21:55:26.387 INFO:teuthology.orchestra.run.smithi112.stdout:5684 write 0x5bea73e thru 0x5beef54 (0x4817 bytes) 2024-09-04T21:55:26.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:26.389 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:26.388+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:26.397 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.398 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.398 INFO:teuthology.orchestra.run.smithi112.stdout:5690 write 0x38aaef7 thru 0x38aced7 (0x1fe1 bytes) 2024-09-04T21:55:26.400 INFO:teuthology.orchestra.run.smithi112.stdout:5692 punch from 0x1357078 to 0x13641eb, (0xd173 bytes) 2024-09-04T21:55:26.400 INFO:teuthology.orchestra.run.smithi112.stdout:5695 punch from 0x435a156 to 0x435d900, (0x37aa bytes) 2024-09-04T21:55:26.400 INFO:teuthology.orchestra.run.smithi112.stdout:5696 read 0x5dbe537 thru 0x5dcd102 (0xebcc bytes) 2024-09-04T21:55:26.401 INFO:teuthology.orchestra.run.smithi112.stdout:5698 write 0x6bd3808 thru 0x6bd92b2 (0x5aab bytes) 2024-09-04T21:55:26.404 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.404 INFO:teuthology.orchestra.run.smithi112.stdout:5700 read 0x5b254f2 thru 0x5b28948 (0x3457 bytes) 2024-09-04T21:55:26.405 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.405 INFO:teuthology.orchestra.run.smithi112.stdout:5702 read 0x44c9c2b thru 0x44cecc4 (0x509a bytes) 2024-09-04T21:55:26.405 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.422 INFO:teuthology.orchestra.run.smithi112.stdout:5704 write 0xacda001 thru 0xace0343 (0x6343 bytes) 2024-09-04T21:55:26.425 INFO:teuthology.orchestra.run.smithi112.stdout:5705 writesame 0xa175652 thru 0xa17f300 data_size 0x1(0x9caf bytes) 2024-09-04T21:55:26.453 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.468 INFO:teuthology.orchestra.run.smithi112.stdout:5707 writesame 0xe09d0f0 thru 0xe0a5c27 data_size 0x6e(0x8b38 bytes) 2024-09-04T21:55:26.468 INFO:teuthology.orchestra.run.smithi112.stdout:5708 read 0x210204d thru 0x210540a (0x33be bytes) 2024-09-04T21:55:26.470 INFO:teuthology.orchestra.run.smithi112.stdout:5709 trunc from 0xe0a5c28 to 0xd57bd7c 2024-09-04T21:55:26.490 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:26.488+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:26.492 INFO:teuthology.orchestra.run.smithi112.stdout:5710 trunc from 0xd57bd7c to 0x28badb6 2024-09-04T21:55:26.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:26.536 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:26.547 INFO:teuthology.orchestra.run.smithi112.stdout:5711 writesame 0x2dfefa4 thru 0x2e0011d data_size 0x1(0x117a bytes) 2024-09-04T21:55:26.549 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.549 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:26.551 INFO:teuthology.orchestra.run.smithi112.stdout:5716 trunc from 0x2e0011e to 0x159c086 2024-09-04T21:55:26.587 INFO:teuthology.orchestra.run.smithi112.stdout:5718 write 0x99d8207 thru 0x99e6052 (0xde4c bytes) 2024-09-04T21:55:26.591 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:26.588+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:26.592 INFO:teuthology.orchestra.run.smithi112.stdout:5721 read 0x2e49752 thru 0x2e566f7 (0xcfa6 bytes) 2024-09-04T21:55:26.593 INFO:teuthology.orchestra.run.smithi112.stdout:5722 read 0x4d1684e thru 0x4d1dd7c (0x752f bytes) 2024-09-04T21:55:26.594 INFO:teuthology.orchestra.run.smithi112.stdout:5723 punch from 0x5f2e4cc to 0x5f36e27, (0x895b bytes) 2024-09-04T21:55:26.594 INFO:teuthology.orchestra.run.smithi112.stdout:5724 read 0x13679b7 thru 0x136f159 (0x77a3 bytes) 2024-09-04T21:55:26.595 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.595 INFO:teuthology.orchestra.run.smithi112.stdout:5726 writesame 0xe7694d thru 0xe83521 data_size 0x1(0xcbd5 bytes) 2024-09-04T21:55:26.644 INFO:teuthology.orchestra.run.smithi112.stdout:5727 read 0x197ea44 thru 0x198679c (0x7d59 bytes) 2024-09-04T21:55:26.646 INFO:teuthology.orchestra.run.smithi112.stdout:5728 flatten 2024-09-04T21:55:26.654 INFO:teuthology.orchestra.run.smithi112.stdout:5731 read 0x3d97584 thru 0x3da410d (0xcb8a bytes) 2024-09-04T21:55:26.658 INFO:teuthology.orchestra.run.smithi112.stdout:5732 trunc from 0x99e6053 to 0xbe5ebfc 2024-09-04T21:55:26.666 INFO:teuthology.orchestra.run.smithi112.stdout:5734 read 0xbdc379e thru 0xbdccc62 (0x94c5 bytes) 2024-09-04T21:55:26.666 INFO:teuthology.orchestra.run.smithi112.stdout:5736 read 0x77867e9 thru 0x7788d75 (0x258d bytes) 2024-09-04T21:55:26.666 INFO:teuthology.orchestra.run.smithi112.stdout:5738 write 0x760044b thru 0x76015b8 (0x116e bytes) 2024-09-04T21:55:26.672 INFO:teuthology.orchestra.run.smithi112.stdout:5740 writesame 0x834b2d2 thru 0x83543ff data_size 0x1(0x912e bytes) 2024-09-04T21:55:26.693 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:26.688+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:26.709 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:26.709 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:26.709 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:26.710 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:26.710 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:26.719 INFO:teuthology.orchestra.run.smithi112.stdout:5741 write 0x490e20d thru 0x4912f5d (0x4d51 bytes) 2024-09-04T21:55:26.721 INFO:teuthology.orchestra.run.smithi112.stdout:5745 read 0xa0d7de5 thru 0xa0d8810 (0xa2c bytes) 2024-09-04T21:55:26.721 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.721 INFO:teuthology.orchestra.run.smithi112.stdout:5748 write 0x4f07636 thru 0x4f15c00 (0xe5cb bytes) 2024-09-04T21:55:26.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:26.725 INFO:teuthology.orchestra.run.smithi112.stdout:5749 read 0x38344d thru 0x385099 (0x1c4d bytes) 2024-09-04T21:55:26.725 INFO:teuthology.orchestra.run.smithi112.stdout:5751 punch from 0x3ee3501 to 0x3ee3c03, (0x702 bytes) 2024-09-04T21:55:26.728 INFO:teuthology.orchestra.run.smithi112.stdout:5752 trunc from 0xbe5ebfc to 0xdbd669c 2024-09-04T21:55:26.735 INFO:teuthology.orchestra.run.smithi112.stdout:5753 writesame 0xb42a8bb thru 0xb42d020 data_size 0x6(0x2766 bytes) 2024-09-04T21:55:26.758 INFO:teuthology.orchestra.run.smithi112.stdout:5754 trunc from 0xdbd669c to 0x3c713d8 2024-09-04T21:55:26.783 INFO:teuthology.orchestra.run.smithi112.stdout:5755 trunc from 0x3c713d8 to 0x699776c 2024-09-04T21:55:26.794 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:26.792+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:26.804 INFO:teuthology.orchestra.run.smithi112.stdout:5757 write 0xa76f136 thru 0xa77d525 (0xe3f0 bytes) 2024-09-04T21:55:26.810 INFO:teuthology.orchestra.run.smithi112.stdout:5758 read 0x6329cca thru 0x632e86c (0x4ba3 bytes) 2024-09-04T21:55:26.810 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.810 INFO:teuthology.orchestra.run.smithi112.stdout:5760 write 0x4908c5f thru 0x490eb00 (0x5ea2 bytes) 2024-09-04T21:55:26.825 INFO:teuthology.orchestra.run.smithi112.stdout:5761 trunc from 0xa77d526 to 0x1918907 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:26.872 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195583756000005, 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:26.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195556971000002, 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.877 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:26.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.881 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195528038000006, 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.882 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195504391, 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195484609999994, 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195465214999999, 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195445942000006, 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.19542672, 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195407883000001, 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195388480000005, 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195367423999997, 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195335245999999, 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195309025, 2024-09-04T21:55:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195280815999993, 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195260775999998, 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195241394000007, 2024-09-04T21:55:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195217806000002, 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195197488999995, 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195177936999997, 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.195158178, 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:26.965 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5762 writesame 0x7cd7770 thru 0x7cdd841 data_size 0x11(0x60d2 bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5763 punch from 0x2c4b146 to 0x2c5410c, (0x8fc6 bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5765 read 0xa46f91 thru 0xa4e68e (0x76fe bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5767 read 0x4b1a553 thru 0x4b2939d (0xee4b bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5768 writesame 0x3490cc3 thru 0x3492c56 data_size 0x4(0x1f94 bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5769 read 0x4ff4a1f thru 0x4ffb605 (0x6be7 bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5772 read 0x2b2bf14 thru 0x2b3bb4a (0xfc37 bytes) 2024-09-04T21:55:26.966 INFO:teuthology.orchestra.run.smithi112.stdout:5773 read 0x333ae16 thru 0x3342103 (0x72ee bytes) 2024-09-04T21:55:26.967 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:26.968 INFO:teuthology.orchestra.run.smithi112.stdout:5775 write 0xb4f4857 thru 0xb4f7398 (0x2b42 bytes) 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5778 writesame 0x5b2aeb3 thru 0x5b2cec1 data_size 0x1(0x200f bytes) 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5779 read 0xfd8250 thru 0xfdcfe0 (0x4d91 bytes) 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5780 read 0x89e5323 thru 0x89edd21 (0x89ff bytes) 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5781 write 0x568212e thru 0x5688fc9 (0x6e9c bytes) 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5782 punch from 0x8653b36 to 0x865bbb8, (0x8082 bytes) 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5783 trunc from 0xb4f7399 to 0x3d80053 2024-09-04T21:55:26.969 INFO:teuthology.orchestra.run.smithi112.stdout:5784 trunc from 0x3d80053 to 0x38106f1 2024-09-04T21:55:26.972 INFO:teuthology.orchestra.run.smithi112.stdout:5785 read 0x1b0e88e thru 0x1b1cbdf (0xe352 bytes) 2024-09-04T21:55:26.972 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:26.968+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:26.988 INFO:teuthology.orchestra.run.smithi112.stdout:5788 write 0x58d817f thru 0x58e63af (0xe231 bytes) 2024-09-04T21:55:27.009 INFO:teuthology.orchestra.run.smithi112.stdout:5789 write 0x9289442 thru 0x929630b (0xceca bytes) 2024-09-04T21:55:27.016 INFO:teuthology.orchestra.run.smithi112.stdout:5790 punch from 0x4aefdec to 0x4af4bf3, (0x4e07 bytes) 2024-09-04T21:55:27.031 INFO:teuthology.orchestra.run.smithi112.stdout:5792 trunc from 0x929630c to 0x2cfe853 2024-09-04T21:55:27.051 INFO:teuthology.orchestra.run.smithi112.stdout:5794 read 0x1a55d78 thru 0x1a61da4 (0xc02d bytes) 2024-09-04T21:55:27.052 INFO:teuthology.orchestra.run.smithi112.stdout:5795 read 0x2a7eb61 thru 0x2a83f19 (0x53b9 bytes) 2024-09-04T21:55:27.071 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:27.068+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:27.083 INFO:teuthology.orchestra.run.smithi112.stdout:5796 write 0xa8b90d0 thru 0xa8bda2c (0x495d bytes) 2024-09-04T21:55:27.090 INFO:teuthology.orchestra.run.smithi112.stdout:5798 read 0x35ee97e thru 0x35efbee (0x1271 bytes) 2024-09-04T21:55:27.091 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.091 INFO:teuthology.orchestra.run.smithi112.stdout:5801 read 0xa4366b8 thru 0xa44315f (0xcaa8 bytes) 2024-09-04T21:55:27.105 INFO:teuthology.orchestra.run.smithi112.stdout:5803 writesame 0xaa05f7e thru 0xaa0c683 data_size 0x1(0x6706 bytes) 2024-09-04T21:55:27.107 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:27.107 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:27.107 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:27.107 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:27.123 INFO:teuthology.orchestra.run.smithi112.stdout:5804 trunc from 0xaa0c684 to 0x4280777 2024-09-04T21:55:27.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:27.171 INFO:teuthology.orchestra.run.smithi112.stdout:5805 write 0xc0d9d05 thru 0xc0dbf83 (0x227f bytes) 2024-09-04T21:55:27.177 INFO:teuthology.orchestra.run.smithi112.stdout:5807 writesame 0x4cd9d22 thru 0x4ce755a data_size 0x1(0xd839 bytes) 2024-09-04T21:55:27.178 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:27.176+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:27.244 INFO:teuthology.orchestra.run.smithi112.stdout:5808 punch from 0x6d934cd to 0x6da044d, (0xcf80 bytes) 2024-09-04T21:55:27.249 INFO:teuthology.orchestra.run.smithi112.stdout:5810 trunc from 0xc0dbf84 to 0xe523fb0 2024-09-04T21:55:27.264 INFO:teuthology.orchestra.run.smithi112.stdout:5811 trunc from 0xe523fb0 to 0xc77269d 2024-09-04T21:55:27.274 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:27.268+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:27.278 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.279 INFO:teuthology.orchestra.run.smithi112.stdout:5813 write 0x49c785f thru 0x49d580c (0xdfae bytes) 2024-09-04T21:55:27.281 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.281 INFO:teuthology.orchestra.run.smithi112.stdout:5821 read 0x289aea thru 0x299a92 (0xffa9 bytes) 2024-09-04T21:55:27.288 INFO:teuthology.orchestra.run.smithi112.stdout:5823 trunc from 0xc77269d to 0x990a265 2024-09-04T21:55:27.292 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:27.292 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:27.293 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:27.293 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:27.293 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:27.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:27.309 INFO:teuthology.orchestra.run.smithi112.stdout:5825 read 0x1841417 thru 0x184b265 (0x9e4f bytes) 2024-09-04T21:55:27.309 INFO:teuthology.orchestra.run.smithi112.stdout:5826 write 0x3d5b2d0 thru 0x3d6af8c (0xfcbd bytes) 2024-09-04T21:55:27.313 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.313 INFO:teuthology.orchestra.run.smithi112.stdout:5828 write 0x9658c40 thru 0x96622ec (0x96ad bytes) 2024-09-04T21:55:27.319 INFO:teuthology.orchestra.run.smithi112.stdout:5830 punch from 0x65ee88 to 0x66c830, (0xd9a8 bytes) 2024-09-04T21:55:27.319 INFO:teuthology.orchestra.run.smithi112.stdout:5831 write 0x333f0c5 thru 0x334b718 (0xc654 bytes) 2024-09-04T21:55:27.327 INFO:teuthology.orchestra.run.smithi112.stdout:5832 trunc from 0x990a265 to 0x5587a89 2024-09-04T21:55:27.347 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.360 INFO:teuthology.orchestra.run.smithi112.stdout:5834 trunc from 0x5587a89 to 0xe725dd0 2024-09-04T21:55:27.374 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:27.368+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:27.387 INFO:teuthology.orchestra.run.smithi112.stdout:5835 trunc from 0xe725dd0 to 0x2820be3 2024-09-04T21:55:27.419 INFO:teuthology.orchestra.run.smithi112.stdout:5836 write 0x7f7bdd6 thru 0x7f87fa1 (0xc1cc bytes) 2024-09-04T21:55:27.425 INFO:teuthology.orchestra.run.smithi112.stdout:5837 punch from 0x1d25d64 to 0x1d26a54, (0xcf0 bytes) 2024-09-04T21:55:27.425 INFO:teuthology.orchestra.run.smithi112.stdout:5839 writesame 0x50bffa7 thru 0x50cf070 data_size 0x77(0xf0ca bytes) 2024-09-04T21:55:27.432 INFO:teuthology.orchestra.run.smithi112.stdout:5840 trunc from 0x7f87fa2 to 0x57b4072 2024-09-04T21:55:27.455 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.600900296999995, 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:55:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:55:27.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:55:27.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.474853034000006, 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.462 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:55:27.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:55:27.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.467 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.278299395999994, 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:55:27.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:55:27.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.250909898000003, 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:55:27.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:55:27.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.227645426999999, 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.478 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:55:27.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:55:27.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.199887637000003, 2024-09-04T21:55:27.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:27.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:55:27.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:55:27.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.199772074999998, 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:55:27.490 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:55:27.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:27.496 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.899597290000003, 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:55:27.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:55:27.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.502 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.740464738, 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:55:27.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:55:27.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:55:27.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:27.508 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.740238680000004, 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:55:27.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.263025612000007, 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.514 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.519 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.033637501000001, 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.520 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:55:27.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:55:27.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.735446859999996, 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.526 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:55:27.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.735334004999999, 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.532 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:55:27.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.725673037000007, 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.538 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:55:27.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.705823312000007, 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:55:27.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.555594626000001, 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.547 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:55:27.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:55:27.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.889267720999996, 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:55:27.553 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:55:27.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:55:27.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:55:27.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.886128534999997, 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.559 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:55:27.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:27.563 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.867817544999994, 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:27.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.567 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:27.568 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:27.568 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:27.569 INFO:teuthology.orchestra.run.smithi112.stdout:5842 write 0xc965121 thru 0xc966104 (0xfe4 bytes) 2024-09-04T21:55:27.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:27.571 INFO:teuthology.orchestra.run.smithi112.stdout:5843 write 0x29cccdb thru 0x29cff7b (0x32a1 bytes) 2024-09-04T21:55:27.571 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.571 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.571 INFO:teuthology.orchestra.run.smithi112.stdout:5846 read 0x5dc9bcb thru 0x5dd6873 (0xcca9 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5847 read 0x50142ae thru 0x501f1e6 (0xaf39 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5848 write 0xa39084a thru 0xa39f08c (0xe843 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5849 read 0x888f1ca thru 0x889ecfe (0xfb35 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5850 write 0x48701b8 thru 0x4874495 (0x42de bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5851 read 0x98aaea2 thru 0x98b9b3f (0xec9e bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5852 write 0x51851db thru 0x5188610 (0x3436 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5853 write 0xb09702c thru 0xb09cd37 (0x5d0c bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5854 write 0x1fe15a9 thru 0x1feee2d (0xd885 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5855 read 0x4e65eb4 thru 0x4e68123 (0x2270 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5856 writesame 0xa666c55 thru 0xa669e8f data_size 0x7(0x323b bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5857 write 0x93f3127 thru 0x94022cc (0xf1a6 bytes) 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5862 trunc from 0xc966105 to 0x4113c90 2024-09-04T21:55:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:5864 write 0x2833c3 thru 0x287165 (0x3da3 bytes) 2024-09-04T21:55:27.573 INFO:teuthology.orchestra.run.smithi112.stdout:5866 clone 34 order 19 su 8192 sc 3 2024-09-04T21:55:27.573 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:27.564+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:27.630 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:55:27.701 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:27.700+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:27.747 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:27.747 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:27.747 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:27.747 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:27.763 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:27.805 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:27.804+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:27.840 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-09-04T21:55:27.864 INFO:teuthology.orchestra.run.smithi112.stdout:5867 writesame 0x7d8fb37 thru 0x7d99483 data_size 0x2f(0x994d bytes) 2024-09-04T21:55:27.872 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:27.868+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:27.879 INFO:teuthology.orchestra.run.smithi112.stdout:5869 writesame 0xb1f8d96 thru 0xb20883d data_size 0x34(0xfaa8 bytes) 2024-09-04T21:55:27.879 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.879 INFO:teuthology.orchestra.run.smithi112.stdout:5872 read 0x8d2ccb5 thru 0x8d375d0 (0xa91c bytes) 2024-09-04T21:55:27.879 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.879 INFO:teuthology.orchestra.run.smithi112.stdout:5874 write 0x1abbde7 thru 0x1ac760a (0xb824 bytes) 2024-09-04T21:55:27.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:27.935 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:27.935 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:27.935 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:27.935 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:27.936 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:27.936 INFO:teuthology.orchestra.run.smithi112.stdout:5877 read 0x1d9c990 thru 0x1da38c4 (0x6f35 bytes) 2024-09-04T21:55:27.938 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.938 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:27.939 INFO:teuthology.orchestra.run.smithi112.stdout:5880 trunc from 0xb20883e to 0xacab971 2024-09-04T21:55:27.949 INFO:teuthology.orchestra.run.smithi112.stdout:5881 writesame 0x8e44a9 thru 0x8efb97 data_size 0x1(0xb6ef bytes) 2024-09-04T21:55:27.958 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:28.007 INFO:teuthology.orchestra.run.smithi112.stdout:5882 write 0xc5c7c1 thru 0xc5ff4c (0x378c bytes) 2024-09-04T21:55:28.007 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:28.004+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:28.060 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:28.060 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:28.073 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:28.068+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:28.077 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:28.077 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:28.081 INFO:teuthology.orchestra.run.smithi112.stdout:5883 read 0x8ef5e61 thru 0x8f0468e (0xe82e bytes) 2024-09-04T21:55:28.081 INFO:teuthology.orchestra.run.smithi112.stdout:5885 read 0x29c8fbe thru 0x29cbba2 (0x2be5 bytes) 2024-09-04T21:55:28.083 INFO:teuthology.orchestra.run.smithi112.stdout:5886 writesame 0x94c2070 thru 0x94c7900 data_size 0x29(0x5891 bytes) 2024-09-04T21:55:28.088 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:28.088 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:28.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131561 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 835584~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.05912994, 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111509741, 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.089 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131561 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027841+0000", 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2899999999999999e-06 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027847+0000", 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8289999999999996e-06 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027868+0000", 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1444999999999999e-05 2024-09-04T21:55:28.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027885+0000", 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6149000000000001e-05 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139001+0000", 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111116231 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139139+0000", 2024-09-04T21:55:28.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013808700000000001 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139350+0000", 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021070999999999999 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131562 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 925696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.093 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0590617120000001, 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111852167, 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131562 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027909+0000", 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.164e-06 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027922+0000", 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2955999999999999e-05 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027942+0000", 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9930999999999999e-05 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027955+0000", 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2916e-05 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139429+0000", 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111473537 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139560+0000", 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013118200000000001 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139760+0000", 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020048099999999999 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131563 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 4022272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0589908189999999, 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112167249, 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.098 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131563 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:28.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028594+0000", 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061550000000000005 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028604+0000", 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3889999999999994e-06 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028627+0000", 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3309e-05 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028644+0000", 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7062e-05 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139847+0000", 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111203128 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139975+0000", 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012794799999999999 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140146+0000", 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017091299999999999 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.103 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131564 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1015808~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0582954740000003, 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11196729499999999, 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131564 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034391+0000", 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005716881 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034402+0000", 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0448999999999999e-05 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034426+0000", 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3975e-05 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034445+0000", 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9386e-05 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140259+0000", 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10581388899999999 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140415+0000", 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015661599999999999 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140642+0000", 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226099 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131577 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2097152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0520050669999996, 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111297805, 2024-09-04T21:55:28.108 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131577 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034966+0000", 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034971+0000", 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0200000000000002e-06 2024-09-04T21:55:28.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035005+0000", 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3763999999999997e-05 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035023+0000", 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8232000000000001e-05 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.145896+0000", 2024-09-04T21:55:28.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.110872963 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146021+0000", 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124888 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146262+0000", 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024174099999999999 2024-09-04T21:55:28.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131578 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2187264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0519230420000003, 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111709911, 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.113 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131578 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:28.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035048+0000", 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3400000000000001e-06 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035054+0000", 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7660000000000004e-06 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035072+0000", 2024-09-04T21:55:28.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8224999999999999e-05 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035085+0000", 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3152999999999999e-05 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146367+0000", 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111282279 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146521+0000", 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015351399999999999 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146757+0000", 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000235634 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.118 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131579 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2277376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0518661729999996, 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11209353800000001, 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131579 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035105+0000", 2024-09-04T21:55:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2470000000000001e-06 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035110+0000", 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.237e-06 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035131+0000", 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1277999999999999e-05 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035153+0000", 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1959000000000001e-05 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146846+0000", 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11169317600000001 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146984+0000", 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137095 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147197+0000", 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000213546 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131580 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2367488~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0517970669999999, 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112535119, 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131580 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042368+0000", 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071953859999999998 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042379+0000", 2024-09-04T21:55:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1069e-05 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042403+0000", 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4158000000000001e-05 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042422+0000", 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8983000000000001e-05 2024-09-04T21:55:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147304+0000", 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10488141500000001 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147478+0000", 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386300000000001 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147708+0000", 2024-09-04T21:55:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023024499999999999 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131596 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 49152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0439003700000002, 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111536857, 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.128 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131596 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043071+0000", 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3799999999999999e-06 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043077+0000", 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043096+0000", 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9681e-05 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043113+0000", 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6416e-05 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154171+0000", 2024-09-04T21:55:28.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111058771 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154353+0000", 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018107400000000001 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154606+0000", 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025366800000000002 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131597 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3809280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0438369139999999, 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111867072, 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.133 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131597 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043134+0000", 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2529999999999999e-06 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043147+0000", 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2693e-05 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043164+0000", 2024-09-04T21:55:28.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043177+0000", 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729e-05 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154692+0000", 2024-09-04T21:55:28.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111515138 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154825+0000", 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013267800000000001 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155000+0000", 2024-09-04T21:55:28.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017526900000000001 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131598 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0437730480000003, 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112278195, 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.138 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131598 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050331+0000", 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071345200000000001 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050342+0000", 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1189000000000001e-05 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050367+0000", 2024-09-04T21:55:28.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4621e-05 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050387+0000", 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819000000000001e-05 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155115+0000", 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104728349 2024-09-04T21:55:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155274+0000", 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015923399999999999 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155475+0000", 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020046300000000001 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131614 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 589824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0358065449999998, 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111567737, 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131614 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051164+0000", 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2160000000000001e-06 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051170+0000", 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2390000000000002e-06 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051187+0000", 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7649e-05 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051200+0000", 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2993000000000001e-05 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162333+0000", 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111132965 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162500+0000", 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016687900000000001 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162731+0000", 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023079599999999999 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131615 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0357508930000003, 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11193702, 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131615 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051220+0000", 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:55:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051225+0000", 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1399999999999999e-06 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051242+0000", 2024-09-04T21:55:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6988999999999999e-05 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051255+0000", 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3261999999999999e-05 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162787+0000", 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111532033 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162923+0000", 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013530399999999999 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163156+0000", 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233095 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131616 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0356963229999998, 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112285617, 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131616 2024-09-04T21:55:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058214+0000", 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069405279999999996 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058226+0000", 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1732999999999999e-05 2024-09-04T21:55:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058253+0000", 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7574999999999999e-05 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058276+0000", 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2334999999999999e-05 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163207+0000", 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104930906 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163336+0000", 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129639 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163559+0000", 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022290100000000001 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131631 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1220608~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.028067815, 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111259443, 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131631 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058903+0000", 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.133e-06 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058909+0000", 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6890000000000001e-06 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058926+0000", 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7166000000000001e-05 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058943+0000", 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7028999999999999e-05 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169875+0000", 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11093157400000001 2024-09-04T21:55:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169996+0000", 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012140499999999999 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170161+0000", 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016544699999999999 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131632 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1613824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0280048859999997, 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111614907, 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131632 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058966+0000", 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058974+0000", 2024-09-04T21:55:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9559999999999997e-06 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059013+0000", 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8688e-05 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059027+0000", 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:55:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170250+0000", 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111222396 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170381+0000", 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013136800000000001 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170580+0000", 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019870700000000001 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131633 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1703936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0279178719999997, 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11190109299999999, 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131633 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059053+0000", 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.353e-06 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059067+0000", 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3482999999999999e-05 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059104+0000", 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7657999999999998e-05 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059129+0000", 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4380999999999999e-05 2024-09-04T21:55:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170639+0000", 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111510427 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170761+0000", 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122302 2024-09-04T21:55:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170953+0000", 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019148900000000001 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131634 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0278179520000004, 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11221152299999999, 2024-09-04T21:55:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131634 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059154+0000", 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059160+0000", 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059199+0000", 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9190999999999998e-05 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059220+0000", 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0499000000000001e-05 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171022+0000", 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11180261599999999 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171158+0000", 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135401 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171363+0000", 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020553999999999999 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131635 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1794048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0277284910000004, 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112472239, 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131635 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066220+0000", 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069786229999999998 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066229+0000", 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066251+0000", 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1829000000000001e-05 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066273+0000", 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2722e-05 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171426+0000", 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10515273 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171548+0000", 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012224 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171714+0000", 2024-09-04T21:55:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000165289 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131652 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2785280~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0198382029999999, 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111334216, 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131652 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074601+0000", 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074699279999999998 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074617+0000", 2024-09-04T21:55:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5882000000000001e-05 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074651+0000", 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4100999999999998e-05 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074674+0000", 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2858999999999999e-05 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178090+0000", 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10341562999999999 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178247+0000", 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156673 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178466+0000", 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021914300000000001 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:28.193 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:28.193 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:28.194 INFO:teuthology.orchestra.run.smithi112.stdout:5889 punch from 0x80eb406 to 0x80ec52a, (0x1124 bytes) 2024-09-04T21:55:28.194 INFO:teuthology.orchestra.run.smithi112.stdout:5890 read 0x9f880b5 thru 0x9f8e0e6 (0x6032 bytes) 2024-09-04T21:55:28.194 INFO:teuthology.orchestra.run.smithi112.stdout:5891 punch from 0x5e69a21 to 0x5e7910c, (0xf6eb bytes) 2024-09-04T21:55:28.194 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:28.195 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:28.195 INFO:teuthology.orchestra.run.smithi112.stdout:5892 trunc from 0xacab971 to 0x42c6425 2024-09-04T21:55:28.196 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:28.197 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:28.198 INFO:teuthology.orchestra.run.smithi112.stdout:5893 read 0x28c9cef thru 0x28d24bd (0x87cf bytes) 2024-09-04T21:55:28.198 INFO:teuthology.orchestra.run.smithi112.stdout:5894 punch from 0x351e00c to 0x351e092, (0x86 bytes) 2024-09-04T21:55:28.198 INFO:teuthology.orchestra.run.smithi112.stdout:5895 write 0x3ebc062 thru 0x3eca01b (0xdfba bytes) 2024-09-04T21:55:28.198 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:28.199 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:28.200 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:28.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:28.201 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:28.191+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:28.270 INFO:teuthology.orchestra.run.smithi112.stdout:5896 trunc from 0x42c6425 to 0x86c445b 2024-09-04T21:55:28.280 INFO:teuthology.orchestra.run.smithi112.stdout:5897 punch from 0x50ce286 to 0x50cf2ca, (0x1044 bytes) 2024-09-04T21:55:28.280 INFO:teuthology.orchestra.run.smithi112.stdout:5898 writesame 0x4094999 thru 0x409d2b6 data_size 0x1(0x891e bytes) 2024-09-04T21:55:28.295 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:28.292+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:28.339 INFO:teuthology.orchestra.run.smithi112.stdout:5899 read 0x8cbf84 thru 0x8da76c (0xe7e9 bytes) 2024-09-04T21:55:28.345 INFO:teuthology.orchestra.run.smithi112.stdout:5901 trunc from 0x86c445b to 0xb4adc3e 2024-09-04T21:55:28.359 INFO:teuthology.orchestra.run.smithi112.stdout:5905 read 0x5a98339 thru 0x5a9e6e5 (0x63ad bytes) 2024-09-04T21:55:28.361 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:28.362 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:28.375 INFO:teuthology.orchestra.run.smithi112.stdout:5907 write 0xd0ef889 thru 0xd0fdfe1 (0xe759 bytes) 2024-09-04T21:55:28.377 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:28.382 INFO:teuthology.orchestra.run.smithi112.stdout:5908 write 0xb4a905a thru 0xb4b8ec1 (0xfe68 bytes) 2024-09-04T21:55:28.389 INFO:teuthology.orchestra.run.smithi112.stdout:5909 read 0x73bc9cc thru 0x73bfa2f (0x3064 bytes) 2024-09-04T21:55:28.389 INFO:teuthology.orchestra.run.smithi112.stdout:5910 punch from 0xa77b756 to 0xa7857a8, (0xa052 bytes) 2024-09-04T21:55:28.390 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.390 INFO:teuthology.orchestra.run.smithi112.stdout:5912 read 0x48f05fc thru 0x48fb8ab (0xb2b0 bytes) 2024-09-04T21:55:28.390 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.390 INFO:teuthology.orchestra.run.smithi112.stdout:5915 write 0x78ebc74 thru 0x78ef247 (0x35d4 bytes) 2024-09-04T21:55:28.396 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.396 INFO:teuthology.orchestra.run.smithi112.stdout:5918 write 0xccbd6b6 thru 0xccc55a1 (0x7eec bytes) 2024-09-04T21:55:28.396 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:28.392+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:28.402 INFO:teuthology.orchestra.run.smithi112.stdout:5919 writesame 0x6a0b8cd thru 0x6a103a6 data_size 0x1a(0x4ada bytes) 2024-09-04T21:55:28.409 INFO:teuthology.orchestra.run.smithi112.stdout:5920 read 0x9b6a32e thru 0x9b702d3 (0x5fa6 bytes) 2024-09-04T21:55:28.409 INFO:teuthology.orchestra.run.smithi112.stdout:5921 writesame 0x37e1ed2 thru 0x37e890e data_size 0x1(0x6a3d bytes) 2024-09-04T21:55:28.452 INFO:teuthology.orchestra.run.smithi112.stdout:5922 read 0x5698e68 thru 0x569cb87 (0x3d20 bytes) 2024-09-04T21:55:28.452 INFO:teuthology.orchestra.run.smithi112.stdout:5923 write 0x844d01e thru 0x8456199 (0x917c bytes) 2024-09-04T21:55:28.458 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.458 INFO:teuthology.orchestra.run.smithi112.stdout:5925 read 0x7c88c77 thru 0x7c8e446 (0x57d0 bytes) 2024-09-04T21:55:28.461 INFO:teuthology.orchestra.run.smithi112.stdout:5926 trunc from 0xd0fdfe2 to 0xe6ed375 2024-09-04T21:55:28.483 INFO:teuthology.orchestra.run.smithi112.stdout:5927 trunc from 0xe6ed375 to 0x6e72a88 2024-09-04T21:55:28.496 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:28.491+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:28.505 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.518 INFO:teuthology.orchestra.run.smithi112.stdout:5929 trunc from 0x6e72a88 to 0xe399499 2024-09-04T21:55:28.529 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:28.529 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:28.530 INFO:teuthology.orchestra.run.smithi112.stdout:5930 read 0x42ff06 thru 0x432565 (0x2660 bytes) 2024-09-04T21:55:28.545 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:28.547 INFO:teuthology.orchestra.run.smithi112.stdout:5931 writesame 0xe8a9870 thru 0xe8ae2ab data_size 0x2(0x4a3c bytes) 2024-09-04T21:55:28.547 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.547 INFO:teuthology.orchestra.run.smithi112.stdout:5934 write 0x60c34fa thru 0x60cf2d2 (0xbdd9 bytes) 2024-09-04T21:55:28.558 INFO:teuthology.orchestra.run.smithi112.stdout:5935 write 0x1dc114a thru 0x1dce3d5 (0xd28c bytes) 2024-09-04T21:55:28.598 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:28.592+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:28.651 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:55:28.671 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:28.671 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:28.676 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.676 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:28.690 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:28.691 INFO:teuthology.orchestra.run.smithi112.stdout:5938 trunc from 0xe8ae2ac to 0x43122b5 2024-09-04T21:55:28.698 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:28.695+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:28.735 INFO:teuthology.orchestra.run.smithi112.stdout:5940 write 0xde8e13a thru 0xde9269a (0x4561 bytes) 2024-09-04T21:55:28.743 INFO:teuthology.orchestra.run.smithi112.stdout:5941 writesame 0x4fac45e thru 0x4fb6008 data_size 0x1(0x9bab bytes) 2024-09-04T21:55:28.757 INFO:teuthology.orchestra.run.smithi112.stdout:5942 writesame 0x1072699 thru 0x107ef09 data_size 0x17(0xc871 bytes) 2024-09-04T21:55:28.799 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:28.795+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:28.826 INFO:teuthology.orchestra.run.smithi112.stdout:5943 read 0x3b921d5 thru 0x3ba01af (0xdfdb bytes) 2024-09-04T21:55:28.827 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:28.827 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:28.830 INFO:teuthology.orchestra.run.smithi112.stdout:5944 write 0x5e21be8 thru 0x5e2f4ea (0xd903 bytes) 2024-09-04T21:55:28.839 INFO:teuthology.orchestra.run.smithi112.stdout:5945 read 0x8a7cbb6 thru 0x8a813ee (0x4839 bytes) 2024-09-04T21:55:28.839 INFO:teuthology.orchestra.run.smithi112.stdout:5946 read 0x4feb11f thru 0x4ff2fc5 (0x7ea7 bytes) 2024-09-04T21:55:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:5949 read 0x16dd9ff thru 0x16ed47b (0xfa7d bytes) 2024-09-04T21:55:28.843 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:28.844 INFO:teuthology.orchestra.run.smithi112.stdout:5950 writesame 0xb988f0 thru 0xba39e8 data_size 0x5(0xb0f9 bytes) 2024-09-04T21:55:28.900 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:28.895+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:28.964 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:28.965 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:28.976 INFO:teuthology.orchestra.run.smithi112.stdout:5951 read 0xca2e65d thru 0xca376ca (0x906e bytes) 2024-09-04T21:55:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:5953 read 0x6ebfaf4 thru 0x6ec1ef4 (0x2401 bytes) 2024-09-04T21:55:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:5954 punch from 0xcc6751 to 0xcd058f, (0x9e3e bytes) 2024-09-04T21:55:28.979 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:28.988 INFO:teuthology.orchestra.run.smithi112.stdout:5955 trunc from 0xde9269b to 0x9536f3d 2024-09-04T21:55:29.001 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:28.995+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:29.027 INFO:teuthology.orchestra.run.smithi112.stdout:5956 writesame 0xb6cb13d thru 0xb6d80a8 data_size 0x3c(0xcf6c bytes) 2024-09-04T21:55:29.027 INFO:teuthology.orchestra.run.smithi112.stdout:5957 write 0x2f53998 thru 0x2f58700 (0x4d69 bytes) 2024-09-04T21:55:29.074 INFO:teuthology.orchestra.run.smithi112.stdout:5958 write 0x679265d thru 0x67a1831 (0xf1d5 bytes) 2024-09-04T21:55:29.101 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:29.096+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:29.102 INFO:teuthology.orchestra.run.smithi112.stdout:5960 write 0xc698d66 thru 0xc69bf74 (0x320f bytes) 2024-09-04T21:55:29.111 INFO:teuthology.orchestra.run.smithi112.stdout:5961 write 0xc6659c5 thru 0xc674dcb (0xf407 bytes) 2024-09-04T21:55:29.116 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:29.116 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:29.119 INFO:teuthology.orchestra.run.smithi112.stdout:5962 writesame 0x2854ac1 thru 0x2860196 data_size 0x57(0xb6d6 bytes) 2024-09-04T21:55:29.132 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:29.179 INFO:teuthology.orchestra.run.smithi112.stdout:5964 read 0x39fc9cb thru 0x3a02caf (0x62e5 bytes) 2024-09-04T21:55:29.182 INFO:teuthology.orchestra.run.smithi112.stdout:5965 read 0xbc91660 thru 0xbc92691 (0x1032 bytes) 2024-09-04T21:55:29.182 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:29.182 INFO:teuthology.orchestra.run.smithi112.stdout:5967 read 0x2582986 thru 0x258e699 (0xbd14 bytes) 2024-09-04T21:55:29.185 INFO:teuthology.orchestra.run.smithi112.stdout:5968 write 0x58a0d5b thru 0x58a7bd0 (0x6e76 bytes) 2024-09-04T21:55:29.196 INFO:teuthology.orchestra.run.smithi112.stdout:5969 read 0x66408de thru 0x6648e50 (0x8573 bytes) 2024-09-04T21:55:29.196 INFO:teuthology.orchestra.run.smithi112.stdout:5970 read 0x801640a thru 0x8020cf1 (0xa8e8 bytes) 2024-09-04T21:55:29.196 INFO:teuthology.orchestra.run.smithi112.stdout:5972 read 0x243b799 thru 0x2446e95 (0xb6fd bytes) 2024-09-04T21:55:29.199 INFO:teuthology.orchestra.run.smithi112.stdout:5976 punch from 0x75a8f84 to 0x75aa83e, (0x18ba bytes) 2024-09-04T21:55:29.199 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:29.199 INFO:teuthology.orchestra.run.smithi112.stdout:5980 read 0xd71dc8 thru 0xd7b78e (0x99c7 bytes) 2024-09-04T21:55:29.200 INFO:teuthology.orchestra.run.smithi112.stdout:5982 write 0x7ef0255 thru 0x7eff99d (0xf749 bytes) 2024-09-04T21:55:29.223 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:29.220+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:29.228 INFO:teuthology.orchestra.run.smithi112.stdout:5983 write 0xe6c8853 thru 0xe6d55cf (0xcd7d bytes) 2024-09-04T21:55:29.264 INFO:teuthology.orchestra.run.smithi112.stdout:5984 trunc from 0xe6d55d0 to 0x3ea1d9e 2024-09-04T21:55:29.293 INFO:teuthology.orchestra.run.smithi112.stdout:compare and write disabled 2024-09-04T21:55:29.301 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:29.301 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:29.302 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:29.296+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:29.316 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:29.317 INFO:teuthology.orchestra.run.smithi112.stdout:5986 write 0x80fb417 thru 0x8100e9d (0x5a87 bytes) 2024-09-04T21:55:29.331 INFO:teuthology.orchestra.run.smithi112.stdout:5987 read 0x3deffb4 thru 0x3dfb672 (0xb6bf bytes) 2024-09-04T21:55:29.339 INFO:teuthology.orchestra.run.smithi112.stdout:5989 trunc from 0x8100e9e to 0x5da4292 2024-09-04T21:55:29.360 INFO:teuthology.orchestra.run.smithi112.stdout:5990 read 0x1aa5411 thru 0x1ab020b (0xadfb bytes) 2024-09-04T21:55:29.361 INFO:teuthology.orchestra.run.smithi112.stdout:5993 read 0x5214cb1 thru 0x521f5df (0xa92f bytes) 2024-09-04T21:55:29.361 INFO:teuthology.orchestra.run.smithi112.stdout:5995 writesame 0x112916b thru 0x11302c1 data_size 0x7(0x7157 bytes) 2024-09-04T21:55:29.403 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:29.399+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:29.451 INFO:teuthology.orchestra.run.smithi112.stdout:5997 writesame 0xa149598 thru 0xa14d3fd data_size 0xe(0x3e66 bytes) 2024-09-04T21:55:29.457 INFO:teuthology.orchestra.run.smithi112.stdout:5998 trunc from 0xa14d3fe to 0xc03a793 2024-09-04T21:55:29.470 INFO:teuthology.orchestra.run.smithi112.stdout:5999 write 0x4eb4b10 thru 0x4ebf982 (0xae73 bytes) 2024-09-04T21:55:29.481 INFO:teuthology.orchestra.run.smithi112.stdout:6000 read 0xbfc5fe9 thru 0xbfcc5be (0x65d6 bytes) 2024-09-04T21:55:29.494 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:29.494 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:29.503 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:29.499+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:29.509 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:29.512 INFO:teuthology.orchestra.run.smithi112.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-09-04T21:55:29.604 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:29.600+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:29.627 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:29.628 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:29.642 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:29.705 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:29.699+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:29.806 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:29.803+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:29.811 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:29.811 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:29.826 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:29.907 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:29.903+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:29.961 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:29.961 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:29.976 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:30.019 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:30.016+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:30.097 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:30.097 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:30.109 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:30.104+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:30.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:30.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:30.243 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:30.243 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:30.243 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:30.239+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:30.258 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:30.342 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:30.342 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:30.342 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:30.339+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:30.358 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:30.359 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:30.375 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:30.376 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:30.392 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:30.392 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:30.392 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:30.411 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:30.404+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:30.413 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:30.413 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:30.421 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:30.421 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:30.434 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:30.434 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:30.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:30.455 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:30.456 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:30.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:30.512 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:30.508+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:30.574 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:30.574 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:30.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896410884999995, 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.575 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:30.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896384100000006, 2024-09-04T21:55:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896355166999996, 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896331520000004, 2024-09-04T21:55:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896311738999998, 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896292344000003, 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896273070999996, 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896253849000004, 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896235012000005, 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896215608999995, 2024-09-04T21:55:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896194553000001, 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896162375000003, 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896136154000004, 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896107944999997, 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896087905000002, 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896068522999997, 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896044935000006, 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896024617999998, 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.896005066000001, 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.895985307000004, 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:30.666 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:30.666 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7] out_osds: [1, 0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-04T21:55:30.666 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-04T21:55:30.667 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-04T21:55:30.667 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-04T21:55:30.668 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:30.669 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:30.664+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:30.670 INFO:tasks.ceph.osd.4:Started 2024-09-04T21:55:30.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:30.672 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:55:30.769 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:30.763+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:0 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:1 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:2 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:abort 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:assert 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:config diff 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:config diff get 2024-09-04T21:55:30.800 INFO:teuthology.orchestra.run.smithi157.stderr:config get 2024-09-04T21:55:30.801 INFO:teuthology.orchestra.run.smithi157.stderr:config help [] 2024-09-04T21:55:30.801 INFO:teuthology.orchestra.run.smithi157.stderr:config set ... 2024-09-04T21:55:30.801 INFO:teuthology.orchestra.run.smithi157.stderr:admin_socket: invalid command 2024-09-04T21:55:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:30.803 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-04T21:55:30.806 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:30.806 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:30.806 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:30.806 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:30.821 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:30.910 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:30.907+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:30.972 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:30.967+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:30.980 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:30.980 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:30.980 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:30.980 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:30.981 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:30.995 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:31.072 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:31.068+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:31.172 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.316560176999999, 2024-09-04T21:55:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:55:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:55:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:55:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:55:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:55:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:55:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:55:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:55:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:55:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:55:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:55:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:55:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:55:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:55:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:55:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:55:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:55:31.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:55:31.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:55:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:55:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:55:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:55:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:55:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:55:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.190512913999996, 2024-09-04T21:55:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:55:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:55:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:55:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:55:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:55:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:55:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:55:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:55:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:55:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:55:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:55:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:55:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:55:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:55:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:55:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:55:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.993959275999998, 2024-09-04T21:55:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:55:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:55:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:55:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:55:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:55:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:55:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:55:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:55:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:55:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:55:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:55:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:55:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:55:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:55:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:55:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:55:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:55:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:55:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:55:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:55:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:55:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:1538 3.1a e28/22 3:5bcb3600:::rbd_data.1188e7c968d9.0000000000000007:head v 28'15, mlcod=27'13)", 2024-09-04T21:55:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.966569777999993, 2024-09-04T21:55:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035820470000000001, 2024-09-04T21:55:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201712+0000", 2024-09-04T21:55:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.201713+0000", 2024-09-04T21:55:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3599999999999999e-06 2024-09-04T21:55:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202429+0000", 2024-09-04T21:55:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071534099999999998 2024-09-04T21:55:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202458+0000", 2024-09-04T21:55:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9450000000000001e-05 2024-09-04T21:55:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202475+0000", 2024-09-04T21:55:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:55:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202587+0000", 2024-09-04T21:55:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111233 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.202623+0000", 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5982999999999998e-05 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205186+0000", 2024-09-04T21:55:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025638610000000002 2024-09-04T21:55:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.205294+0000", 2024-09-04T21:55:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107693 2024-09-04T21:55:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.943305307000003, 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:55:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:55:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:55:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:55:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:55:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:55:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:55:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:55:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:55:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:55:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:55:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:55:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:55:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:55:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:55:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:55:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:55:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.915547516999993, 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:55:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.915431955000003, 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:55:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:55:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1788 3.19 3:9f4017eb:::rbd_data.1188e7c968d9.00000000000001bc:head [set-alloc-hint object_size 524288 write_size 524288,write 196608~4393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.615257170000007, 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035520999999999999, 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1788 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553025+0000", 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553027+0000", 2024-09-04T21:55:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0930000000000001e-06 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553038+0000", 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1103e-05 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553052+0000", 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4256999999999999e-05 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553060+0000", 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7440000000000007e-06 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553128+0000", 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8099e-05 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553309+0000", 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018105399999999999 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.553405+0000", 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5966999999999998e-05 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556168+0000", 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027629569999999999 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556492+0000", 2024-09-04T21:55:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032429799999999999 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556533+0000", 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1155999999999997e-05 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.556577+0000", 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3371999999999997e-05 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.456124618000004, 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:55:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:55:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.455898559999994, 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:55:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.978685491999997, 2024-09-04T21:55:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:55:31.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:31.274 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.749297381000005, 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:55:31.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.45110674, 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.281 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:55:31.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.450993885000003, 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:55:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7648 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:5 [sparse-read 196608~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.441332916999997, 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003494867, 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7648 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726949+0000", 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726950+0000", 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1930000000000001e-06 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726956+0000", 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8889999999999999e-06 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726975+0000", 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8858000000000001e-05 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.726989+0000", 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3808999999999999e-05 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729352+0000", 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023634720000000001 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.729605+0000", 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025267900000000001 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.730444+0000", 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083896700000000003 2024-09-04T21:55:31.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7680 3.8 e31/22 3:122f710e:::rbd_data.1188b1294ada.0000000000000004:head v 31'245, mlcod=31'243)", 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.421483191999997, 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003509215, 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746799+0000", 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.746802+0000", 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7999999999999999e-06 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747413+0000", 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061112899999999997 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747461+0000", 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8183999999999998e-05 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747484+0000", 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2892e-05 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747652+0000", 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016845099999999999 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.747714+0000", 2024-09-04T21:55:31.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1653000000000001e-05 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750222+0000", 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025077060000000002 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.750308+0000", 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6399999999999999e-05 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.271254506000005, 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.306 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:31.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:55:31.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:55:31.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:55:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 82.604927601, 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:55:31.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:55:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7864 3.8 e32/22 3:174ae6c3:::rbd_data.1188b1294ada.000000000000005e:head v 32'250, mlcod=31'247)", 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 82.601788415000001, 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003563587, 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566493+0000", 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566496+0000", 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7530000000000002e-06 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566515+0000", 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8630999999999998e-05 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566547+0000", 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1813000000000002e-05 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.566572+0000", 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4881000000000001e-05 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567803+0000", 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012313140000000001 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.567866+0000", 2024-09-04T21:55:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3527000000000003e-05 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569922+0000", 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020552349999999999 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.570057+0000", 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013543299999999999 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 82.583477424999998, 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:55:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:55:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.329 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:31.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:31.331 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:31.328+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:31.431 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:31.427+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:31.458 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:31.458 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:31.458 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:31.458 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:31.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:31.532 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:31.528+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:31.633 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:31.627+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:31.633 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:31.634 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:31.648 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:31.654 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:55:31.734 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:31.728+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:31.748 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:31.744+0000 7f6381b6f7c0 -1 Falling back to public interface 2024-09-04T21:55:31.782 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131561 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 835584~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7524368700000004, 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111509741, 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131561 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027841+0000", 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2899999999999999e-06 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027847+0000", 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8289999999999996e-06 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027868+0000", 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1444999999999999e-05 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027885+0000", 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6149000000000001e-05 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139001+0000", 2024-09-04T21:55:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111116231 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139139+0000", 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013808700000000001 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139350+0000", 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021070999999999999 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131562 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 925696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7523686420000004, 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111852167, 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131562 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027909+0000", 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.164e-06 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027922+0000", 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2955999999999999e-05 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027942+0000", 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9930999999999999e-05 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027955+0000", 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2916e-05 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139429+0000", 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111473537 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139560+0000", 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013118200000000001 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139760+0000", 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020048099999999999 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131563 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 4022272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7522977490000002, 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112167249, 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131563 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028594+0000", 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061550000000000005 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028604+0000", 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3889999999999994e-06 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028627+0000", 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3309e-05 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028644+0000", 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7062e-05 2024-09-04T21:55:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139847+0000", 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111203128 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139975+0000", 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012794799999999999 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140146+0000", 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017091299999999999 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131564 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1015808~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7516024039999998, 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11196729499999999, 2024-09-04T21:55:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131564 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034391+0000", 2024-09-04T21:55:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005716881 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034402+0000", 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0448999999999999e-05 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034426+0000", 2024-09-04T21:55:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3975e-05 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034445+0000", 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9386e-05 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140259+0000", 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10581388899999999 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140415+0000", 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015661599999999999 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140642+0000", 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226099 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131577 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2097152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7453119969999999, 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111297805, 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131577 2024-09-04T21:55:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034966+0000", 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034971+0000", 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0200000000000002e-06 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035005+0000", 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3763999999999997e-05 2024-09-04T21:55:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035023+0000", 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8232000000000001e-05 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.145896+0000", 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.110872963 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146021+0000", 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124888 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146262+0000", 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024174099999999999 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131578 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2187264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7452299720000006, 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111709911, 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131578 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035048+0000", 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3400000000000001e-06 2024-09-04T21:55:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035054+0000", 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7660000000000004e-06 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035072+0000", 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8224999999999999e-05 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035085+0000", 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3152999999999999e-05 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146367+0000", 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111282279 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146521+0000", 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015351399999999999 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146757+0000", 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000235634 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131579 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2277376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7451731029999991, 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11209353800000001, 2024-09-04T21:55:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131579 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:31.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035105+0000", 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2470000000000001e-06 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035110+0000", 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.237e-06 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035131+0000", 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1277999999999999e-05 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035153+0000", 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1959000000000001e-05 2024-09-04T21:55:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146846+0000", 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11169317600000001 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146984+0000", 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137095 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147197+0000", 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000213546 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131580 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2367488~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7451039969999993, 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112535119, 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.818 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131580 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042368+0000", 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071953859999999998 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042379+0000", 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1069e-05 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042403+0000", 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4158000000000001e-05 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042422+0000", 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8983000000000001e-05 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147304+0000", 2024-09-04T21:55:31.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10488141500000001 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147478+0000", 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386300000000001 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147708+0000", 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023024499999999999 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131596 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 49152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7372072999999997, 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111536857, 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.824 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131596 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043071+0000", 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3799999999999999e-06 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043077+0000", 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043096+0000", 2024-09-04T21:55:31.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9681e-05 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043113+0000", 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6416e-05 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154171+0000", 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111058771 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154353+0000", 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018107400000000001 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154606+0000", 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025366800000000002 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131597 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3809280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7371438440000002, 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111867072, 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131597 2024-09-04T21:55:31.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043134+0000", 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2529999999999999e-06 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043147+0000", 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2693e-05 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043164+0000", 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043177+0000", 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729e-05 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154692+0000", 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111515138 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154825+0000", 2024-09-04T21:55:31.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013267800000000001 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155000+0000", 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017526900000000001 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131598 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:31.833 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7370799780000006, 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112278195, 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131598 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:31.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050331+0000", 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071345200000000001 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050342+0000", 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1189000000000001e-05 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050367+0000", 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4621e-05 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050387+0000", 2024-09-04T21:55:31.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819000000000001e-05 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155115+0000", 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104728349 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155274+0000", 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015923399999999999 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155475+0000", 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020046300000000001 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131614 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 589824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7291134750000001, 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111567737, 2024-09-04T21:55:31.838 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131614 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051164+0000", 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2160000000000001e-06 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051170+0000", 2024-09-04T21:55:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2390000000000002e-06 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051187+0000", 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7649e-05 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051200+0000", 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2993000000000001e-05 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162333+0000", 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111132965 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162500+0000", 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016687900000000001 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162731+0000", 2024-09-04T21:55:31.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023079599999999999 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131615 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7290578229999998, 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11193702, 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.843 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131615 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:31.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051220+0000", 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051225+0000", 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1399999999999999e-06 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051242+0000", 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6988999999999999e-05 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051255+0000", 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3261999999999999e-05 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162787+0000", 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111532033 2024-09-04T21:55:31.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162923+0000", 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013530399999999999 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163156+0000", 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233095 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131616 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7290032530000001, 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112285617, 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131616 2024-09-04T21:55:31.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058214+0000", 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069405279999999996 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058226+0000", 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1732999999999999e-05 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058253+0000", 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7574999999999999e-05 2024-09-04T21:55:31.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058276+0000", 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2334999999999999e-05 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163207+0000", 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104930906 2024-09-04T21:55:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163336+0000", 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129639 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163559+0000", 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022290100000000001 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131631 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1220608~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7213747450000003, 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111259443, 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131631 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.853 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058903+0000", 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.133e-06 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058909+0000", 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6890000000000001e-06 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058926+0000", 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7166000000000001e-05 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058943+0000", 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7028999999999999e-05 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169875+0000", 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11093157400000001 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169996+0000", 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012140499999999999 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170161+0000", 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016544699999999999 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131632 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1613824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7213118160000001, 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111614907, 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131632 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058966+0000", 2024-09-04T21:55:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058974+0000", 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9559999999999997e-06 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059013+0000", 2024-09-04T21:55:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8688e-05 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059027+0000", 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170250+0000", 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111222396 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170381+0000", 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013136800000000001 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170580+0000", 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019870700000000001 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131633 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1703936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7212248020000001, 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11190109299999999, 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131633 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059053+0000", 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.353e-06 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059067+0000", 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3482999999999999e-05 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059104+0000", 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7657999999999998e-05 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059129+0000", 2024-09-04T21:55:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4380999999999999e-05 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170639+0000", 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111510427 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170761+0000", 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122302 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170953+0000", 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019148900000000001 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131634 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7211248819999998, 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11221152299999999, 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.868 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131634 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059154+0000", 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059160+0000", 2024-09-04T21:55:31.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059199+0000", 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9190999999999998e-05 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059220+0000", 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0499000000000001e-05 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171022+0000", 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11180261599999999 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171158+0000", 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135401 2024-09-04T21:55:31.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171363+0000", 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020553999999999999 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131635 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1794048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7210354209999998, 2024-09-04T21:55:31.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112472239, 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131635 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:31.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066220+0000", 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069786229999999998 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066229+0000", 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066251+0000", 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1829000000000001e-05 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066273+0000", 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2722e-05 2024-09-04T21:55:31.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171426+0000", 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10515273 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171548+0000", 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012224 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171714+0000", 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000165289 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131652 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2785280~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:31.878 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7131451329999994, 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111334216, 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131652 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:31.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074601+0000", 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074699279999999998 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074617+0000", 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5882000000000001e-05 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074651+0000", 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4100999999999998e-05 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074674+0000", 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2858999999999999e-05 2024-09-04T21:55:31.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178090+0000", 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10341562999999999 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178247+0000", 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156673 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178466+0000", 2024-09-04T21:55:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021914300000000001 2024-09-04T21:55:31.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.883 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:31.883 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:31.883 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:31.884 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:31.885 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:31.880+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:31.985 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:31.979+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:32.013 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:32.013 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:32.013 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:32.013 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:32.028 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:32.085 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:32.080+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:32.186 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:32.183+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:32.196 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:32.196 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:32.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:32.301 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:32.299+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:32.377 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 61.872190472, 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.378 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:32.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527414086, 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.383 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527385481000003, 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.388 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527365443999997, 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527346221999998, 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527327431000003, 2024-09-04T21:55:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:32.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527295488999997, 2024-09-04T21:55:32.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:32.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527266959999999, 2024-09-04T21:55:32.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:32.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:32.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527247349, 2024-09-04T21:55:32.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:32.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:32.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527229056000003, 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:32.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.427 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527206241999998, 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.428 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527187105000003, 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.437 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527166438999998, 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:32.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:32.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.442 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527135661999999, 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:32.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:32.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527116294000002, 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527092334999999, 2024-09-04T21:55:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527045086000001, 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.527019408000001, 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.526985381999999, 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 46.526960981000002, 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:32.476 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:32.477 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:32.478 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:32.479 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:32.475+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:32.579 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:32.575+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:32.609 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:32.609 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:32.609 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:32.609 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:32.624 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:32.669 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:32.670 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:32.680 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:32.675+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:32.688 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:32.689 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:32.707 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:32.707 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:32.725 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:32.725 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:32.725 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:32.746 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:32.746 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:32.754 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:32.754 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:32.769 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:32.771 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:32.771 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:32.780 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:32.775+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:32.790 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:32.791 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:32.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:32.882 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:32.879+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:32.896 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:32.896 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:32.910 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:32.983 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:32.979+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:33.026 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.026 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.040 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:33.091 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:33.088+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:33.163 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.163 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.177 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:33.184 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:33.179+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:33.285 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:33.280+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:33.338 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.338 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.352 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:33.385 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:33.383+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:33.487 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:33.483+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:33.517 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.517 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.531 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:33.588 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:33.584+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:33.655 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.655 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.669 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:33.684 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:55:33.730 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:33.727+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:33.790 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:33.787+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:33.841 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.842 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:33.901 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:33.899+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:33.975 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:33.975 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:33.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:34.034 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:34.031+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:34.091 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:34.087+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:34.150 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:34.151 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:34.166 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:34.210 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:34.207+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:34.294 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:34.288+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:34.331 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:34.332 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:34.345 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:34.395 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:34.391+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:34.474 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:34.474 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:34.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:34.530 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:34.527+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:34.597 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:34.591+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:34.652 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:34.652 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:34.666 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:55:34.719 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:34.715+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:34.796 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:34.797 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:34.792+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:34.830 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:34.830 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:34.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:34.898 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:34.892+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:34.952 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:34.969 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:34.969 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:34.987 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:34.988 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:34.999 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:34.995+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:35.004 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:35.010 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:35.010 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:35.010 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:35.010 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:35.024 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:35.025 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:35.025 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:35.046 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:35.047 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:35.067 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:35.067 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:35.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:35.113 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:35.111+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:35.140 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:35.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:35.140 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:35.140 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:35.140 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:35.154 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:35.201 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:35.200+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:35.302 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:35.299+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:35.322 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:35.322 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:35.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:35.322 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131561 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 835584~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.292213672000001, 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111509741, 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131561 2024-09-04T21:55:35.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027841+0000", 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2899999999999999e-06 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027847+0000", 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8289999999999996e-06 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027868+0000", 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1444999999999999e-05 2024-09-04T21:55:35.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027885+0000", 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6149000000000001e-05 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139001+0000", 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111116231 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139139+0000", 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013808700000000001 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139350+0000", 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021070999999999999 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131562 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 925696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.327 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.292145443999999, 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111852167, 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131562 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027909+0000", 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.164e-06 2024-09-04T21:55:35.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027922+0000", 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2955999999999999e-05 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027942+0000", 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9930999999999999e-05 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027955+0000", 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2916e-05 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139429+0000", 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111473537 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139560+0000", 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013118200000000001 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139760+0000", 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020048099999999999 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131563 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 4022272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.292074551000001, 2024-09-04T21:55:35.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112167249, 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131563 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:35.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028594+0000", 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061550000000000005 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028604+0000", 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3889999999999994e-06 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028627+0000", 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3309e-05 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028644+0000", 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7062e-05 2024-09-04T21:55:35.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139847+0000", 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111203128 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139975+0000", 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012794799999999999 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140146+0000", 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017091299999999999 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131564 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1015808~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.291379206, 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11196729499999999, 2024-09-04T21:55:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131564 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034391+0000", 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005716881 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034402+0000", 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0448999999999999e-05 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034426+0000", 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3975e-05 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034445+0000", 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9386e-05 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140259+0000", 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10581388899999999 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140415+0000", 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015661599999999999 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140642+0000", 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226099 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131577 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2097152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.285088799, 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111297805, 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131577 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034966+0000", 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034971+0000", 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0200000000000002e-06 2024-09-04T21:55:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035005+0000", 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3763999999999997e-05 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035023+0000", 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8232000000000001e-05 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.145896+0000", 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.110872963 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146021+0000", 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124888 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146262+0000", 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024174099999999999 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131578 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2187264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.285006773999999, 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111709911, 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131578 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:55:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035048+0000", 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3400000000000001e-06 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035054+0000", 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7660000000000004e-06 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035072+0000", 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8224999999999999e-05 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035085+0000", 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3152999999999999e-05 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146367+0000", 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111282279 2024-09-04T21:55:35.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146521+0000", 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015351399999999999 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146757+0000", 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000235634 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131579 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2277376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.284949905, 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11209353800000001, 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131579 2024-09-04T21:55:35.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035105+0000", 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2470000000000001e-06 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035110+0000", 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.237e-06 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035131+0000", 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1277999999999999e-05 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035153+0000", 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1959000000000001e-05 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146846+0000", 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11169317600000001 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146984+0000", 2024-09-04T21:55:35.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137095 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147197+0000", 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000213546 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131580 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2367488~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:35.356 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.284880799, 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112535119, 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131580 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042368+0000", 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071953859999999998 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042379+0000", 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1069e-05 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042403+0000", 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4158000000000001e-05 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042422+0000", 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8983000000000001e-05 2024-09-04T21:55:35.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147304+0000", 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10488141500000001 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147478+0000", 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386300000000001 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147708+0000", 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023024499999999999 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131596 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 49152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.276984102, 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111536857, 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.361 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131596 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043071+0000", 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3799999999999999e-06 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043077+0000", 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:55:35.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043096+0000", 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9681e-05 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043113+0000", 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6416e-05 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154171+0000", 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111058771 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154353+0000", 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018107400000000001 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154606+0000", 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025366800000000002 2024-09-04T21:55:35.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131597 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3809280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.276920646000001, 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111867072, 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131597 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043134+0000", 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2529999999999999e-06 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043147+0000", 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2693e-05 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043164+0000", 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043177+0000", 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729e-05 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154692+0000", 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111515138 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154825+0000", 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013267800000000001 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155000+0000", 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017526900000000001 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131598 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.276856779999999, 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112278195, 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131598 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.371 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050331+0000", 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071345200000000001 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050342+0000", 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1189000000000001e-05 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050367+0000", 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4621e-05 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050387+0000", 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819000000000001e-05 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155115+0000", 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104728349 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155274+0000", 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015923399999999999 2024-09-04T21:55:35.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155475+0000", 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020046300000000001 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131614 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 589824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.268890277000001, 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111567737, 2024-09-04T21:55:35.375 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131614 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:35.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051164+0000", 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2160000000000001e-06 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051170+0000", 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2390000000000002e-06 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051187+0000", 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7649e-05 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051200+0000", 2024-09-04T21:55:35.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2993000000000001e-05 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162333+0000", 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111132965 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162500+0000", 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016687900000000001 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162731+0000", 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023079599999999999 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131615 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.268834625, 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11193702, 2024-09-04T21:55:35.380 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131615 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051220+0000", 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051225+0000", 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1399999999999999e-06 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051242+0000", 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6988999999999999e-05 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051255+0000", 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3261999999999999e-05 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162787+0000", 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111532033 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162923+0000", 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013530399999999999 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163156+0000", 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233095 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131616 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.268780055000001, 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112285617, 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131616 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.388 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058214+0000", 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069405279999999996 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058226+0000", 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1732999999999999e-05 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058253+0000", 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7574999999999999e-05 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058276+0000", 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2334999999999999e-05 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163207+0000", 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104930906 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163336+0000", 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129639 2024-09-04T21:55:35.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163559+0000", 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022290100000000001 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131631 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1220608~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.261151547000001, 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111259443, 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131631 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058903+0000", 2024-09-04T21:55:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.133e-06 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058909+0000", 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6890000000000001e-06 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058926+0000", 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7166000000000001e-05 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058943+0000", 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7028999999999999e-05 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169875+0000", 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11093157400000001 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169996+0000", 2024-09-04T21:55:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012140499999999999 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170161+0000", 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016544699999999999 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131632 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1613824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.261088618, 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111614907, 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131632 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058966+0000", 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058974+0000", 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9559999999999997e-06 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059013+0000", 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8688e-05 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059027+0000", 2024-09-04T21:55:35.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170250+0000", 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111222396 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170381+0000", 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013136800000000001 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170580+0000", 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019870700000000001 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131633 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1703936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.261001604, 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11190109299999999, 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.402 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131633 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059053+0000", 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.353e-06 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059067+0000", 2024-09-04T21:55:35.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3482999999999999e-05 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059104+0000", 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7657999999999998e-05 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059129+0000", 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4380999999999999e-05 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170639+0000", 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111510427 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170761+0000", 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122302 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170953+0000", 2024-09-04T21:55:35.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019148900000000001 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131634 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.260901684, 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11221152299999999, 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.407 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131634 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059154+0000", 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059160+0000", 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059199+0000", 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9190999999999998e-05 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059220+0000", 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0499000000000001e-05 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171022+0000", 2024-09-04T21:55:35.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11180261599999999 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171158+0000", 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135401 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171363+0000", 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020553999999999999 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131635 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1794048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.260812223, 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112472239, 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.412 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131635 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066220+0000", 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069786229999999998 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066229+0000", 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066251+0000", 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1829000000000001e-05 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066273+0000", 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2722e-05 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171426+0000", 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10515273 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171548+0000", 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012224 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171714+0000", 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000165289 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131652 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2785280~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.252921935, 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111334216, 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131652 2024-09-04T21:55:35.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074601+0000", 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074699279999999998 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074617+0000", 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5882000000000001e-05 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074651+0000", 2024-09-04T21:55:35.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4100999999999998e-05 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074674+0000", 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2858999999999999e-05 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178090+0000", 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10341562999999999 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178247+0000", 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156673 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178466+0000", 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021914300000000001 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.421 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:35.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:35.423 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:35.419+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:35.524 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:35.520+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:35.600 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:35.601 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:35.615 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:35.625 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:35.620+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:35.726 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:35.720+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:35.743 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:35.743 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:35.743 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:35.743 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:35.743 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:35.759 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:35.802 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:35.796+0000 7f6381b6f7c0 -1 osd.4 84 log_to_monitors true 2024-09-04T21:55:35.803 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:35.826 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:35.823+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:35.888 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.384077325999996, 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:35.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:35.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:35.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039300939999997, 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.893 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:35.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:35.897 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039272335, 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.898 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:35.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039252298000001, 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.903 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039233076000002, 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.908 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:35.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039214285, 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:35.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:35.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039182343, 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:35.917 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039153814000002, 2024-09-04T21:55:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039134203000003, 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.03911591, 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:35.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.936 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039093096000002, 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039073959, 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:35.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.945 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039053293000002, 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.946 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:35.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:35.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039022516000003, 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:35.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.955 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.039003147999999, 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:35.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:35.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.038979189000003, 2024-09-04T21:55:35.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:35.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.038931939999998, 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:35.965 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:35.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.969 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.038906261999998, 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.970 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.038872236000003, 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:35.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.038847834999999, 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:35.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:35.984 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:no valid command found; 10 closest matches: 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:0 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:1 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:2 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:abort 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:assert 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:bluefs debug_inject_read_zeros 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:bluefs files list 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:bluefs stats 2024-09-04T21:55:35.986 INFO:teuthology.orchestra.run.smithi157.stderr:bluestore allocator dump block 2024-09-04T21:55:35.987 INFO:teuthology.orchestra.run.smithi157.stderr:bluestore allocator fragmentation block 2024-09-04T21:55:35.987 INFO:teuthology.orchestra.run.smithi157.stderr:admin_socket: invalid command 2024-09-04T21:55:35.987 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:35.987 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:35.979+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:35.987 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-04T21:55:36.086 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:36.083+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:36.115 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:36.115 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:36.115 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:36.115 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:36.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:36.187 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:36.180+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:36.251 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:36.251 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:36.251 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:36.251 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:36.251 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:36.265 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:36.288 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:36.283+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:36.388 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:36.383+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.991425148000005, 2024-09-04T21:55:36.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:55:36.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:55:36.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.806884092999994, 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:36.443 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:55:36.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43351 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473588663999998, 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051338846, 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.449 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43351 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4499999999999997e-07 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959605+0000", 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-04T21:55:36.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959613+0000", 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8250000000000005e-06 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959618+0000", 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.417e-06 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010746+0000", 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051127924999999998 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010835+0000", 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9444000000000006e-05 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010941+0000", 2024-09-04T21:55:36.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105312 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43353 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1269760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473564611999997, 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051544344999999998, 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.454 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43353 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959629+0000", 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959635+0000", 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0430000000000004e-06 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959640+0000", 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0320000000000003e-06 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010976+0000", 2024-09-04T21:55:36.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051336248000000001 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011067+0000", 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0826000000000001e-05 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011170+0000", 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103507 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43355 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473542469999998, 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051760575000000003, 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43355 2024-09-04T21:55:36.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7800000000000002e-07 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959651+0000", 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959658+0000", 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5730000000000002e-06 2024-09-04T21:55:36.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959663+0000", 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011205+0000", 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051541489000000003 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011296+0000", 2024-09-04T21:55:36.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1105000000000003e-05 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011409+0000", 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112927 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43358 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:36.463 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473519504999999, 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051974999000000001, 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43358 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9999999999999998e-07 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959674+0000", 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959682+0000", 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9500000000000001e-06 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959687+0000", 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1420000000000001e-06 2024-09-04T21:55:36.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011451+0000", 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051763862000000001 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011538+0000", 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7322000000000002e-05 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011646+0000", 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010772000000000001 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43361 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473496087999997, 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052194391, 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.468 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43361 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959695+0000", 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959697+0000", 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.418e-06 2024-09-04T21:55:36.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959703+0000", 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9989999999999997e-06 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959708+0000", 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.904e-06 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011687+0000", 2024-09-04T21:55:36.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051978506000000001 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011781+0000", 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4301000000000004e-05 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011889+0000", 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107796 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.472 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43362 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473475297, 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052922181999999998, 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43362 2024-09-04T21:55:36.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959716+0000", 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959718+0000", 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959725+0000", 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4660000000000002e-06 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959730+0000", 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1019999999999996e-06 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011924+0000", 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052193145000000003 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012015+0000", 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0969000000000003e-05 2024-09-04T21:55:36.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012637+0000", 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062277700000000003 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43364 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1351680~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473452893999998, 2024-09-04T21:55:36.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052915188000000002, 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43364 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3799999999999997e-07 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959740+0000", 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959747+0000", 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9149999999999999e-06 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959754+0000", 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8000000000000001e-06 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012425+0000", 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052670531 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012525+0000", 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100581 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012653+0000", 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012751700000000001 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.481 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43366 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.473429246000002, 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052927226000000001, 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.482 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43366 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959762+0000", 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0399999999999996e-07 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959764+0000", 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3949999999999999e-06 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960370+0000", 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060537999999999998 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960389+0000", 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9326000000000001e-05 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012680+0000", 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052291232 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012688+0000", 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3890000000000005e-06 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.486 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43368 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.472770560999997, 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052287419000000002, 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.487 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43368 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960422+0000", 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.84e-06 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960431+0000", 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960971+0000", 2024-09-04T21:55:36.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000539965 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960991+0000", 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0832999999999999e-05 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012703+0000", 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051711615000000002 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012707+0000", 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3499999999999999e-06 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43369 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.472169854999997, 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051701824, 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.491 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43369 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961022+0000", 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6139999999999999e-06 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961030+0000", 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1489999999999994e-06 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961051+0000", 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961065+0000", 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4163e-05 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012718+0000", 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051653533000000001 2024-09-04T21:55:36.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012722+0000", 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43371 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.472104793, 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051652328999999997, 2024-09-04T21:55:36.495 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43371 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961087+0000", 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961106+0000", 2024-09-04T21:55:36.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9097e-05 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961146+0000", 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0281999999999998e-05 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961162+0000", 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5993000000000001e-05 2024-09-04T21:55:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012734+0000", 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051571744000000003 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012738+0000", 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.499 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43373 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.472007085999998, 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051570802999999998, 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.500 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43373 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961185+0000", 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.423e-06 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961191+0000", 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9429999999999997e-06 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961210+0000", 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8943000000000001e-05 2024-09-04T21:55:36.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961223+0000", 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3506000000000001e-05 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012750+0000", 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051527008999999999 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012754+0000", 2024-09-04T21:55:36.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9790000000000004e-06 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43375 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.471947911000001, 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051526660000000002, 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.504 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43375 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961244+0000", 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961249+0000", 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961270+0000", 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0925999999999999e-05 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961283+0000", 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2986000000000001e-05 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012765+0000", 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0514825 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012769+0000", 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8079999999999998e-06 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43377 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.471890676000001, 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051486632999999997, 2024-09-04T21:55:36.508 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43377 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961301+0000", 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961306+0000", 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.69e-06 2024-09-04T21:55:36.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961323+0000", 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6976000000000001e-05 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961336+0000", 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2913e-05 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012781+0000", 2024-09-04T21:55:36.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051444996999999999 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012786+0000", 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9340000000000003e-06 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43379 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:36.512 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.471838030000001, 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051454868000000001, 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43379 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961354+0000", 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0729999999999999e-06 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961358+0000", 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6099999999999999e-06 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961375+0000", 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7091999999999998e-05 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961390+0000", 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4803e-05 2024-09-04T21:55:36.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012803+0000", 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051413343 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012807+0000", 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9469999999999996e-06 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43381 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.471781602, 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051415832000000002, 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43381 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.517 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961410+0000", 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961415+0000", 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.977e-06 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961431+0000", 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6016e-05 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961444+0000", 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2816000000000001e-05 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012821+0000", 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051376809000000002 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012825+0000", 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0910000000000003e-06 2024-09-04T21:55:36.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43383 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.471728474000003, 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051378030999999998, 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.521 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43383 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961463+0000", 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961473+0000", 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961535+0000", 2024-09-04T21:55:36.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2787999999999996e-05 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961555+0000", 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9646000000000001e-05 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012836+0000", 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051281032999999997 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012840+0000", 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9840000000000003e-06 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43385 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.471600527, 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051265590999999999, 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.525 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43385 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961591+0000", 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961597+0000", 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961615+0000", 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7836e-05 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961628+0000", 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2976000000000001e-05 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012851+0000", 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051223109000000003 2024-09-04T21:55:36.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012856+0000", 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7870000000000001e-06 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.529 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:36.531 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:36.531 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:36.527+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:36.632 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:36.627+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:36.658 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:36.658 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:36.658 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:36.658 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:36.672 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:36.720 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:55:36.732 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:36.728+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:36.751 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:36.748+0000 7f637594a640 -1 osd.4 84 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-04T21:55:36.803 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:36.804 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:36.804 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:36.804 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:36.804 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:36.819 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:36.833 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:36.831+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:36.934 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:36.928+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:36.991 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.627260632000002, 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:36.992 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:36.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:55:36.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:36.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49175 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 475136~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194564206999999, 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021719833000000001, 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:36.998 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49175 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:36.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793193+0000", 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4e-07 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793196+0000", 2024-09-04T21:55:37.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793206+0000", 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4229999999999994e-06 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793213+0000", 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4370000000000001e-06 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814388+0000", 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021174454999999998 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814604+0000", 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021587900000000001 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814912+0000", 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030848600000000001 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49290 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 40960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194533544000002, 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022433502000000001, 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.003 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49290 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793224+0000", 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6599999999999996e-07 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793227+0000", 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793240+0000", 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-05 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793252+0000", 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2843000000000001e-05 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815053+0000", 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021800657000000001 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815301+0000", 2024-09-04T21:55:37.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024764399999999999 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815656+0000", 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035569200000000001 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49297 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 106496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194492852000003, 2024-09-04T21:55:37.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022960876000000002, 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49297 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:37.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793268+0000", 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4460000000000001e-06 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793277+0000", 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2299999999999997e-06 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793284+0000", 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9530000000000002e-06 2024-09-04T21:55:37.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815785+0000", 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022500857999999999 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815996+0000", 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211065 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816224+0000", 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022851699999999999 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49303 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 172032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194463030000001, 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023578159000000001, 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.012 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49303 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793294+0000", 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793297+0000", 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9500000000000001e-06 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793306+0000", 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2140000000000002e-06 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793313+0000", 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:55:37.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816342+0000", 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023028835000000001 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816580+0000", 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023813 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816872+0000", 2024-09-04T21:55:37.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029135000000000001 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49309 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 237568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194392067999999, 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024080300999999998, 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.017 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49309 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:37.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793365+0000", 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.054e-06 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793374+0000", 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793388+0000", 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3403e-05 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793396+0000", 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8830000000000006e-06 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816958+0000", 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023561992 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817170+0000", 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021269799999999999 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817445+0000", 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027435799999999999 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49326 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 303104~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.021 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194317146000003, 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024582155000000001, 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49326 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793440+0000", 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.003e-06 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793445+0000", 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6840000000000004e-06 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793464+0000", 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9085000000000001e-05 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793478+0000", 2024-09-04T21:55:37.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3716999999999999e-05 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817523+0000", 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024045704000000001 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817738+0000", 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021439899999999999 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818021+0000", 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028356300000000002 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49332 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194265027, 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025083616, 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.026 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49332 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793492+0000", 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793496+0000", 2024-09-04T21:55:37.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793506+0000", 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9729999999999992e-06 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793516+0000", 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6269999999999995e-06 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818099+0000", 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024583880999999998 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818307+0000", 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020786200000000001 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818575+0000", 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026768400000000002 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.030 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49338 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194230841, 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025617277000000001, 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.031 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49338 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5600000000000005e-07 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793533+0000", 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1930000000000004e-06 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793542+0000", 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0669999999999998e-06 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793552+0000", 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1290000000000002e-06 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818661+0000", 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025109056000000001 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818870+0000", 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000208929 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819143+0000", 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027324700000000001 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.035 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49344 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.194194856000003, 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026178261000000001, 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49344 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:37.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793565+0000", 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8930000000000001e-06 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793574+0000", 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793581+0000", 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8399999999999997e-06 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819234+0000", 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025653441999999999 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819463+0000", 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022879500000000001 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819740+0000", 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027665399999999998 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51043 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860657685, 2024-09-04T21:55:37.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022470637000000002, 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51043 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:37.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127100+0000", 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127114+0000", 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4017e-05 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127128+0000", 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4005000000000001e-05 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127139+0000", 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1007e-05 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148542+0000", 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021403575000000001 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148727+0000", 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018442900000000001 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149569+0000", 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084262400000000004 2024-09-04T21:55:37.044 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51044 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 262144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860603079000001, 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023138729, 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.045 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51044 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127154+0000", 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127158+0000", 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.172e-06 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127194+0000", 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5222000000000003e-05 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127206+0000", 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2159e-05 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149626+0000", 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022420476000000002 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149779+0000", 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015296299999999999 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150292+0000", 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051277400000000002 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.049 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51045 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 360448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860532530999997, 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023446379, 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51045 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.050 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127225+0000", 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127230+0000", 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4110000000000004e-06 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127245+0000", 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4714e-05 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127256+0000", 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0787e-05 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150350+0000", 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023094001999999999 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150502+0000", 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151805 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150670+0000", 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016861 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51046 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860484204000002, 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023742831999999998, 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.054 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51046 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127273+0000", 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.23e-07 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127277+0000", 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9650000000000002e-06 2024-09-04T21:55:37.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127291+0000", 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3549e-05 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127302+0000", 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1270000000000001e-05 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150727+0000", 2024-09-04T21:55:37.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023424996999999999 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150882+0000", 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015530200000000001 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151015+0000", 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013292600000000001 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51047 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 557056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.86044107, 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024031869000000001, 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51047 2024-09-04T21:55:37.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127316+0000", 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5300000000000003e-07 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127320+0000", 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0749999999999999e-06 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127334+0000", 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3587000000000001e-05 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127344+0000", 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0461e-05 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151070+0000", 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023725311999999998 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151219+0000", 2024-09-04T21:55:37.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014918599999999999 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151347+0000", 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012839500000000001 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51048 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860398592999999, 2024-09-04T21:55:37.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024346703000000001, 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51048 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:37.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127359+0000", 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.85e-07 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127375+0000", 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6084000000000001e-05 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127389+0000", 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.412e-05 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127400+0000", 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-04T21:55:37.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151402+0000", 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024002787000000001 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151573+0000", 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170941 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151705+0000", 2024-09-04T21:55:37.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131161 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51049 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860329147000002, 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024613729000000001, 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.068 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51049 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:37.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127428+0000", 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6700000000000002e-07 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127433+0000", 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0000000000000004e-06 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127447+0000", 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4224e-05 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127459+0000", 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1112e-05 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151761+0000", 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024302397999999999 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151914+0000", 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015300399999999999 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152041+0000", 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127024 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51050 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860282843999997, 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024957087999999999, 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51050 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:55:37.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127479+0000", 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7160000000000002e-06 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127494+0000", 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4997000000000001e-05 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127506+0000", 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.168e-05 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152104+0000", 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024597810000000001 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152273+0000", 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016888100000000001 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152431+0000", 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000158097 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51051 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.860235349999996, 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025255452000000001, 2024-09-04T21:55:37.077 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51051 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127522+0000", 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127527+0000", 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6299999999999997e-06 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127541+0000", 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4225e-05 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127552+0000", 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0971999999999999e-05 2024-09-04T21:55:37.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152484+0000", 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024931803999999998 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152641+0000", 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015691699999999999 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152776+0000", 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013595400000000001 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:106971 3.0 e82/38 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head v 82'257, mlcod=81'254)", 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.357941021999999, 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045768883000000003, 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:37.082 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629817+0000", 2024-09-04T21:55:37.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9400000000000001e-06 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629834+0000", 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6246999999999999e-05 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629860+0000", 2024-09-04T21:55:37.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6628999999999999e-05 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629873+0000", 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3216e-05 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629999+0000", 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012590799999999999 2024-09-04T21:55:37.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.630064+0000", 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4546000000000002e-05 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675498+0000", 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045433922000000002 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:37.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675584+0000", 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6475000000000001e-05 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:37.087 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:37.088 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:37.089 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:37.087+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:37.089 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:37.189 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:37.183+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:37.234 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:37.234 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:37.250 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:37.250 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:37.260 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:37.260 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:37.261 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:37.261 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:37.266 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:37.267 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:37.274 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:37.283 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:37.283 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:37.283 INFO:teuthology.orchestra.run.smithi112.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-04T21:55:37.289 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:37.287+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:37.301 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:37.301 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:37.319 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:37.319 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:37.336 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:37.336 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-04T21:55:37.342 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-04T21:55:37.391 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:37.388+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:37.397 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:37.397 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:37.412 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:37.492 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:37.487+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:37.575 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:37.575 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:37.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:37.592 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:37.588+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:37.693 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:37.692+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:37.709 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:55:37.736 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:37.736 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:37.755 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:37.793 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:37.788+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:37.894 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:37.888+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:37.916 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:37.916 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:37.930 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:37.995 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:37.991+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:38.052 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:38.052 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:38.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:38.110 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:38.107+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:38.198 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:38.192+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:38.231 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:38.232 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:38.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:38.299 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:38.295+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:38.369 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:38.369 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:38.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:38.426 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:38.423+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:38.500 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:38.495+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:38.546 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:38.546 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:38.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:38.606 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:38.603+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:38.701 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:38.699+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:38.728 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:38.728 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:38.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:38.802 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:38.796+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:38.903 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:38.903 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:38.903 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:38.899+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:38.916 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:39.003 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:39.000+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:39.075 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:39.075 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:39.089 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:39.104 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:39.100+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:39.205 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:39.199+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:39.211 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:39.211 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:39.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:39.306 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:39.300+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:39.343 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:39.398 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:39.398 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:39.406 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:39.403+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:39.413 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:39.476 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:39.476 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:39.493 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:39.493 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:39.507 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:39.503+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:39.509 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:39.509 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:39.524 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:39.525 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:39.545 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:39.546 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-04T21:55:39.566 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:39.566 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.065302388999996, 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:55:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720526003000003, 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720497397999999, 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:55:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:55:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720477361, 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:55:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720458139000002, 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:55:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:55:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720439347999999, 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:55:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720407406, 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:55:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:55:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720378877000002, 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:55:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:55:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720359266000003, 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:55:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:55:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720340972999999, 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:55:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:55:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720318159000001, 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:55:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720299021999999, 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:55:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:55:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720278356000001, 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:55:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720247579000002, 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720228210999998, 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:55:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720204252000002, 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:55:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720157002999997, 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:55:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720131324999997, 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:55:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720097299000003, 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:55:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:55:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.720072897999998, 2024-09-04T21:55:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:39.660 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:39.661 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:39.662 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:39.662 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:39.662 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:39.663 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:39.663 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:39.656+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:39.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:55:39.786 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:39.783+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:39.819 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:39.820 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:39.820 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:39.820 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:39.839 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:39.883 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:39.880+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:39.964 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:39.960+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:39.991 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:39.991 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:39.991 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:40.007 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:40.091 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:40.088+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:40.166 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:40.166 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:40.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:40.166 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:40.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.721163719000003, 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:55:40.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:55:40.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.536622664000006, 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.173 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:55:40.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:55:40.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43351 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203327235000003, 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051338846, 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43351 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:40.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4499999999999997e-07 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959605+0000", 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959613+0000", 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8250000000000005e-06 2024-09-04T21:55:40.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959618+0000", 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.417e-06 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010746+0000", 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051127924999999998 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010835+0000", 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9444000000000006e-05 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010941+0000", 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105312 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.183 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43353 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1269760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203303183000003, 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051544344999999998, 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43353 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.184 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:40.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959629+0000", 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959635+0000", 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0430000000000004e-06 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959640+0000", 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0320000000000003e-06 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010976+0000", 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051336248000000001 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011067+0000", 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0826000000000001e-05 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011170+0000", 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103507 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43355 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203281040999997, 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051760575000000003, 2024-09-04T21:55:40.188 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43355 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7800000000000002e-07 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959651+0000", 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-04T21:55:40.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959658+0000", 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5730000000000002e-06 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959663+0000", 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011205+0000", 2024-09-04T21:55:40.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051541489000000003 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011296+0000", 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1105000000000003e-05 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011409+0000", 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112927 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43358 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203258075999997, 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051974999000000001, 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43358 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.193 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:40.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9999999999999998e-07 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959674+0000", 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959682+0000", 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9500000000000001e-06 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959687+0000", 2024-09-04T21:55:40.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1420000000000001e-06 2024-09-04T21:55:40.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011451+0000", 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051763862000000001 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011538+0000", 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7322000000000002e-05 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011646+0000", 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010772000000000001 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43361 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203234659000003, 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052194391, 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.198 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43361 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959695+0000", 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959697+0000", 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.418e-06 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959703+0000", 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9989999999999997e-06 2024-09-04T21:55:40.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959708+0000", 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.904e-06 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011687+0000", 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051978506000000001 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011781+0000", 2024-09-04T21:55:40.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4301000000000004e-05 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011889+0000", 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107796 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43362 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203213867999999, 2024-09-04T21:55:40.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052922181999999998, 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43362 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959716+0000", 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959718+0000", 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959725+0000", 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4660000000000002e-06 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959730+0000", 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1019999999999996e-06 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011924+0000", 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052193145000000003 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012015+0000", 2024-09-04T21:55:40.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0969000000000003e-05 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012637+0000", 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062277700000000003 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43364 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1351680~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203191465000003, 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052915188000000002, 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43364 2024-09-04T21:55:40.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3799999999999997e-07 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959740+0000", 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959747+0000", 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9149999999999999e-06 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959754+0000", 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8000000000000001e-06 2024-09-04T21:55:40.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012425+0000", 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052670531 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012525+0000", 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100581 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012653+0000", 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012751700000000001 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43366 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.213 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.203167817000001, 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052927226000000001, 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43366 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.214 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959762+0000", 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0399999999999996e-07 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959764+0000", 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3949999999999999e-06 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960370+0000", 2024-09-04T21:55:40.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060537999999999998 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960389+0000", 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9326000000000001e-05 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012680+0000", 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052291232 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012688+0000", 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3890000000000005e-06 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43368 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:40.218 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.202509132000003, 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052287419000000002, 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43368 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960422+0000", 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.84e-06 2024-09-04T21:55:40.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960431+0000", 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960971+0000", 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000539965 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960991+0000", 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0832999999999999e-05 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012703+0000", 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051711615000000002 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012707+0000", 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3499999999999999e-06 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43369 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201908426000003, 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051701824, 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.223 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43369 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961022+0000", 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6139999999999999e-06 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961030+0000", 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1489999999999994e-06 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961051+0000", 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961065+0000", 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4163e-05 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012718+0000", 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051653533000000001 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012722+0000", 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43371 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201843363999998, 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051652328999999997, 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.227 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43371 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961087+0000", 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961106+0000", 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9097e-05 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961146+0000", 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0281999999999998e-05 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961162+0000", 2024-09-04T21:55:40.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5993000000000001e-05 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012734+0000", 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051571744000000003 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012738+0000", 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.231 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43373 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201745656999996, 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051570802999999998, 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.232 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43373 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961185+0000", 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.423e-06 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961191+0000", 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9429999999999997e-06 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961210+0000", 2024-09-04T21:55:40.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8943000000000001e-05 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961223+0000", 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3506000000000001e-05 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012750+0000", 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051527008999999999 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012754+0000", 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9790000000000004e-06 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43375 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201686482, 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051526660000000002, 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.236 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43375 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961244+0000", 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961249+0000", 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:55:40.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961270+0000", 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0925999999999999e-05 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961283+0000", 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2986000000000001e-05 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012765+0000", 2024-09-04T21:55:40.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0514825 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012769+0000", 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8079999999999998e-06 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43377 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201629247, 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051486632999999997, 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43377 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:40.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961301+0000", 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961306+0000", 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.69e-06 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961323+0000", 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6976000000000001e-05 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961336+0000", 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2913e-05 2024-09-04T21:55:40.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012781+0000", 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051444996999999999 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012786+0000", 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9340000000000003e-06 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.244 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43379 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201576600999999, 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051454868000000001, 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.245 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43379 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:40.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961354+0000", 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0729999999999999e-06 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961358+0000", 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6099999999999999e-06 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961375+0000", 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7091999999999998e-05 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961390+0000", 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4803e-05 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012803+0000", 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051413343 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012807+0000", 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9469999999999996e-06 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43381 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201520172999999, 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051415832000000002, 2024-09-04T21:55:40.249 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43381 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:40.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961410+0000", 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961415+0000", 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.977e-06 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961431+0000", 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6016e-05 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961444+0000", 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2816000000000001e-05 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012821+0000", 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051376809000000002 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012825+0000", 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0910000000000003e-06 2024-09-04T21:55:40.253 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43383 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201467045000001, 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051378030999999998, 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.254 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43383 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:40.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961463+0000", 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961473+0000", 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961535+0000", 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2787999999999996e-05 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961555+0000", 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9646000000000001e-05 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012836+0000", 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051281032999999997 2024-09-04T21:55:40.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012840+0000", 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9840000000000003e-06 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43385 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.201339097999998, 2024-09-04T21:55:40.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051265590999999999, 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43385 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:40.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961591+0000", 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961597+0000", 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961615+0000", 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7836e-05 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961628+0000", 2024-09-04T21:55:40.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2976000000000001e-05 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012851+0000", 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051223109000000003 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012856+0000", 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7870000000000001e-06 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.263 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.263 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.263 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:40.263 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:40.163+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:40.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:40.267 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:40.263+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:40.359 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:55:40.368 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:40.364+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:40.444 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:40.444 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:40.444 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:40.444 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:40.458 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:40.469 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:40.463+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:40.569 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:40.564+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:40.623 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:40.623 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:40.623 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:40.623 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:40.623 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:40.637 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:40.671 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:40.667+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:40.771 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:40.767+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.441603727, 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:55:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:55:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49175 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 475136~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008907301999997, 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021719833000000001, 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49175 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793193+0000", 2024-09-04T21:55:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4e-07 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793196+0000", 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793206+0000", 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4229999999999994e-06 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793213+0000", 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4370000000000001e-06 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814388+0000", 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021174454999999998 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814604+0000", 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021587900000000001 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814912+0000", 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030848600000000001 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49290 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 40960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008876639, 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022433502000000001, 2024-09-04T21:55:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49290 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793224+0000", 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6599999999999996e-07 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793227+0000", 2024-09-04T21:55:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793240+0000", 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-05 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793252+0000", 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2843000000000001e-05 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815053+0000", 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021800657000000001 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815301+0000", 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024764399999999999 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815656+0000", 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035569200000000001 2024-09-04T21:55:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49297 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 106496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008835947000001, 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022960876000000002, 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49297 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793268+0000", 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4460000000000001e-06 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793277+0000", 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2299999999999997e-06 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793284+0000", 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9530000000000002e-06 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815785+0000", 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022500857999999999 2024-09-04T21:55:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815996+0000", 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211065 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816224+0000", 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022851699999999999 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49303 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 172032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008806125, 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023578159000000001, 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49303 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793294+0000", 2024-09-04T21:55:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793297+0000", 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9500000000000001e-06 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793306+0000", 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2140000000000002e-06 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793313+0000", 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816342+0000", 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023028835000000001 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816580+0000", 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023813 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816872+0000", 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029135000000000001 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49309 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 237568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008735162999997, 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024080300999999998, 2024-09-04T21:55:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49309 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793365+0000", 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.054e-06 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793374+0000", 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793388+0000", 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3403e-05 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793396+0000", 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8830000000000006e-06 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816958+0000", 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023561992 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817170+0000", 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021269799999999999 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817445+0000", 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027435799999999999 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49326 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 303104~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008660241000001, 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024582155000000001, 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49326 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793440+0000", 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.003e-06 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793445+0000", 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6840000000000004e-06 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793464+0000", 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9085000000000001e-05 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793478+0000", 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3716999999999999e-05 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817523+0000", 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024045704000000001 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817738+0000", 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021439899999999999 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818021+0000", 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028356300000000002 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49332 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008608121999998, 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025083616, 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49332 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793492+0000", 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793496+0000", 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793506+0000", 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9729999999999992e-06 2024-09-04T21:55:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793516+0000", 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6269999999999995e-06 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818099+0000", 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024583880999999998 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818307+0000", 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020786200000000001 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818575+0000", 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026768400000000002 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49338 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008573935999998, 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025617277000000001, 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49338 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5600000000000005e-07 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793533+0000", 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1930000000000004e-06 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793542+0000", 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0669999999999998e-06 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793552+0000", 2024-09-04T21:55:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1290000000000002e-06 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818661+0000", 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025109056000000001 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818870+0000", 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000208929 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819143+0000", 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027324700000000001 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49344 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.008537951000001, 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026178261000000001, 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49344 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793565+0000", 2024-09-04T21:55:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8930000000000001e-06 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793574+0000", 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793581+0000", 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8399999999999997e-06 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819234+0000", 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025653441999999999 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819463+0000", 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022879500000000001 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819740+0000", 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027665399999999998 2024-09-04T21:55:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51043 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.675000779999998, 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022470637000000002, 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51043 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127100+0000", 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127114+0000", 2024-09-04T21:55:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4017e-05 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127128+0000", 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4005000000000001e-05 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127139+0000", 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1007e-05 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148542+0000", 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021403575000000001 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148727+0000", 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018442900000000001 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149569+0000", 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084262400000000004 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51044 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 262144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674946173999999, 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023138729, 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51044 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127154+0000", 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127158+0000", 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.172e-06 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127194+0000", 2024-09-04T21:55:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5222000000000003e-05 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127206+0000", 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2159e-05 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149626+0000", 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022420476000000002 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149779+0000", 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015296299999999999 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150292+0000", 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051277400000000002 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51045 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 360448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674875626000002, 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023446379, 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51045 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127225+0000", 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127230+0000", 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4110000000000004e-06 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127245+0000", 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4714e-05 2024-09-04T21:55:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127256+0000", 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0787e-05 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150350+0000", 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023094001999999999 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150502+0000", 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151805 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150670+0000", 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016861 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51046 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674827299, 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023742831999999998, 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51046 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127273+0000", 2024-09-04T21:55:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.23e-07 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127277+0000", 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9650000000000002e-06 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127291+0000", 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3549e-05 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127302+0000", 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1270000000000001e-05 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150727+0000", 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023424996999999999 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150882+0000", 2024-09-04T21:55:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015530200000000001 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151015+0000", 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013292600000000001 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51047 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 557056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674784164999998, 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024031869000000001, 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51047 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127316+0000", 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5300000000000003e-07 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127320+0000", 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0749999999999999e-06 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127334+0000", 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3587000000000001e-05 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127344+0000", 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0461e-05 2024-09-04T21:55:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151070+0000", 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023725311999999998 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151219+0000", 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014918599999999999 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151347+0000", 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012839500000000001 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51048 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674741687999997, 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024346703000000001, 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51048 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127359+0000", 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.85e-07 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127375+0000", 2024-09-04T21:55:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6084000000000001e-05 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127389+0000", 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.412e-05 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127400+0000", 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151402+0000", 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024002787000000001 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151573+0000", 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170941 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151705+0000", 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131161 2024-09-04T21:55:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51049 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674672242, 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024613729000000001, 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51049 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127428+0000", 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6700000000000002e-07 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127433+0000", 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0000000000000004e-06 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127447+0000", 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4224e-05 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127459+0000", 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1112e-05 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151761+0000", 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024302397999999999 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151914+0000", 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015300399999999999 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152041+0000", 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127024 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51050 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674625939000002, 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024957087999999999, 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51050 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127479+0000", 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7160000000000002e-06 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127494+0000", 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4997000000000001e-05 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127506+0000", 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.168e-05 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152104+0000", 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024597810000000001 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152273+0000", 2024-09-04T21:55:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016888100000000001 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152431+0000", 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000158097 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51051 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.674578445000002, 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025255452000000001, 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51051 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127522+0000", 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127527+0000", 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6299999999999997e-06 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127541+0000", 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4225e-05 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127552+0000", 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0971999999999999e-05 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152484+0000", 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024931803999999998 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152641+0000", 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015691699999999999 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152776+0000", 2024-09-04T21:55:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013595400000000001 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:106971 3.0 e82/38 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head v 82'257, mlcod=81'254)", 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 32.172284116999997, 2024-09-04T21:55:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045768883000000003, 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629817+0000", 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9400000000000001e-06 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629834+0000", 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6246999999999999e-05 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629860+0000", 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6628999999999999e-05 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629873+0000", 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3216e-05 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629999+0000", 2024-09-04T21:55:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012590799999999999 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.630064+0000", 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4546000000000002e-05 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675498+0000", 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045433922000000002 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675584+0000", 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6475000000000001e-05 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:40.903 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:40.904 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:40.905 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:40.899+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:40.989 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:41.005 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:41.003+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:41.031 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.031 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143697 3.13 3.ae45cd73 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.66069369499999997, 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.66071835099999998, 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for peered", 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143697 2024-09-04T21:55:41.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370766+0000", 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7429999999999998e-06 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370776+0000", 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8549999999999997e-06 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370795+0000", 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9599e-05 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370807+0000", 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1652e-05 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370896+0000", 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104000000000006e-05 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370911+0000", 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4986e-05 2024-09-04T21:55:41.035 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.036 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.036 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.036 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.036 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-04T21:55:41.036 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 1 2024-09-04T21:55:41.036 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:41.046 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:41.106 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:41.103+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:41.119 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.119 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:41.119 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:41.119 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:41.134 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-04T21:55:41.207 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:41.203+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:41.208 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.208 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:41.208 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:41.208 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:41.209 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:41.224 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:41.309 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:41.303+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:41.345 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.726791168999995, 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.715785925999995, 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36764 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.172103152000005, 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066796242000000006, 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36764 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170047+0000", 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170048+0000", 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170051+0000", 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6429999999999999e-06 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170058+0000", 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4070000000000004e-06 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170064+0000", 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.485e-06 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236594+0000", 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066530086000000002 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236652+0000", 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236844+0000", 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000191423 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.172079175999997, 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.172038490000006, 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.172005091000003, 2024-09-04T21:55:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171961820000007, 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171917139000001, 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171855983, 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171825941999998, 2024-09-04T21:55:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171781171000006, 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171732879000004, 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171697234000007, 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171448018000007, 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.171409148999999, 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.166262403999994, 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.166219144999999, 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.166175273999997, 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.166138996000001, 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.166112889999994, 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.450 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.450 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.450 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:41.451 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:41.452 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:41.447+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:41.552 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:41.547+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:41.583 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.583 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:41.583 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:41.583 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:41.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:41.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:41.683 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:41.679+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:41.754 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:41.751+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:41.763 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.763 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:41.763 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:41.763 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:41.764 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:41.779 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:41.819 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:41.835 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:41.835 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:41.851 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:41.852 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:41.867 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:41.863+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:be1dd030:::rbd_data.1188af11ea53.0000000000000038:20, version: 90'432, data_included: [458752~8192,516096~8192], data_size: 16384, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:be1dd030:::rbd_data.1188af11ea53.0000000000000038:20@90'432, size: 524288, copy_subset: [(0, 524288)], clone_subset: {}, snapset: 20=[]:{20: [20]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 524288, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.785358+0000", 2024-09-04T21:55:41.948 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.1591439819999998, 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026534699999999998, 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785358+0000", 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785358+0000", 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785359+0000", 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3e-06 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785397+0000", 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7385999999999997e-05 2024-09-04T21:55:41.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785479+0000", 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1903999999999994e-05 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785511+0000", 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1863000000000003e-05 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785700+0000", 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189605 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.785724+0000", 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4015999999999998e-05 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.788012+0000", 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022873960000000001 2024-09-04T21:55:41.952 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23, version: 95'461, data_included: [0~262144,327680~65536,507904~15827,3260416~65536,3407872~32768,3555328~65536,3670016~32768,3850240~32768], data_size: 572883, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23@95'461, size: 3883008, copy_subset: [(0, 3883008)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3883008, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.133176272, 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060935900000000003, 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811327+0000", 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3629999999999999e-06 2024-09-04T21:55:41.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811906+0000", 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057844000000000005 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811940+0000", 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4096e-05 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811967+0000", 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7202000000000001e-05 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812156+0000", 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189073 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812200+0000", 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3748000000000002e-05 2024-09-04T21:55:41.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.817419+0000", 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052196680000000002 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23, version: 95'467, data_included: [2883584~32768,3080192~16384,3145728~65536,3342336~49152,3440640~32768,3620864~49152], data_size: 245760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23@95'467, size: 3670016, copy_subset: [(0, 3670016)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3670016, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.115182978, 2024-09-04T21:55:41.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044989239999999996, 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:41.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829321+0000", 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5999999999999999e-06 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829583+0000", 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026244899999999999 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829622+0000", 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8359000000000003e-05 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829679+0000", 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7102999999999999e-05 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829878+0000", 2024-09-04T21:55:41.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019976299999999999 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829903+0000", 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4604999999999999e-05 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.833818+0000", 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039150449999999998 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.11 106/105 [PushOp(3:8fc5947f:::rbd_data.1188af11ea53.000000000000001e:20, version: 90'451, data_included: [155648~5034,376832~8192], data_size: 13226, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:8fc5947f:::rbd_data.1188af11ea53.000000000000001e:20@90'451, size: 385024, copy_subset: [(0, 385024)], clone_subset: {}, snapset: 20=[]:{20: [20]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 385024, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.850962+0000", 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.0935400629999998, 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025443919999999999, 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.850962+0000", 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.850962+0000", 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.850964+0000", 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7e-06 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.851035+0000", 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1301000000000001e-05 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.851085+0000", 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0164999999999998e-05 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.851132+0000", 2024-09-04T21:55:41.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6578e-05 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.851325+0000", 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000193439 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.851344+0000", 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9000000000000001e-05 2024-09-04T21:55:41.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.853506+0000", 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002162209 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143634 3.9 3:904b5dc8:::rbd_data.11885236e45a.0000000000000079:head [delete] snapc 1d=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:39.860191+0000", 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.0843113770000001, 2024-09-04T21:55:41.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002987459, 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143634 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860191+0000", 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860191+0000", 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860193+0000", 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-04T21:55:41.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860201+0000", 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4009999999999997e-06 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860217+0000", 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6022999999999999e-05 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860227+0000", 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0117e-05 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860306+0000", 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8460000000000004e-05 2024-09-04T21:55:41.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860486+0000", 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000180182 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860587+0000", 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010109499999999999 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:41.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.861302+0000", 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00071482299999999996 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.863123+0000", 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018214240000000001 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.863155+0000", 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2002000000000003e-05 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.863178+0000", 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2687000000000001e-05 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143635 3.1b 3:dc6f77c5:::rbd_data.11885236e45a.000000000000007a:head [delete] snapc 1d=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:39.860255+0000", 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.0842469320000001, 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024722120000000001, 2024-09-04T21:55:41.973 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143635 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860255+0000", 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860255+0000", 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860256+0000", 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9699999999999994e-07 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860261+0000", 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4490000000000001e-06 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860270+0000", 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8840000000000006e-06 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860277+0000", 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860357+0000", 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0259999999999994e-05 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860548+0000", 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019091400000000001 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:41.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.860745+0000", 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019648499999999999 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.862371+0000", 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001626111 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.862634+0000", 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026316300000000001 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.862695+0000", 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0965000000000003e-05 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.862727+0000", 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2323000000000002e-05 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1a 107/106 [PushOp(3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f, version: 88'672, data_included: [0~473726,951144~64182], data_size: 537908, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f@88'672, size: 2861013, copy_subset: [(0, 2861013)], clone_subset: {}, snapset: 1f=[]:{1f: [1f]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2861013, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:39.877701+0000", 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.0668015180000001, 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002657576, 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.877701+0000", 2024-09-04T21:55:41.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.877701+0000", 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.877701+0000", 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.878329+0000", 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062760799999999999 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.878356+0000", 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7331999999999999e-05 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.878392+0000", 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5453000000000001e-05 2024-09-04T21:55:41.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.878500+0000", 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108613 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.878509+0000", 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6370000000000005e-06 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:39.880358+0000", 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018491009999999999 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2959 3.f e108/105 3:f7c4ed88:::rbd_data.1188388e16dc.0000000000000001:1c v 108'438, mlcod=107'435)", 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5798542530000002, 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055309019999999999, 2024-09-04T21:55:41.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364649+0000", 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364676+0000", 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7135e-05 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364688+0000", 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2082000000000001e-05 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364700+0000", 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1885999999999999e-05 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364799+0000", 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9593999999999995e-05 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364831+0000", 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1628000000000003e-05 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370108+0000", 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052768629999999997 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370179+0000", 2024-09-04T21:55:41.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1046999999999999e-05 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.3.0:3938 3.19 e108/105 3:9dde4d2b:::rbd_data.1188388e16dc.0000000000000003:1c v 108'1143, mlcod=106'1139)", 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.368290+0000", 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5762121360000001, 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002735604, 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.989 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368290+0000", 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368290+0000", 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368291+0000", 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7299999999999995e-07 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368298+0000", 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7049999999999999e-06 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368310+0000", 2024-09-04T21:55:41.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1718e-05 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368323+0000", 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2512e-05 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368421+0000", 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8590000000000003e-05 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.368451+0000", 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9853999999999999e-05 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370966+0000", 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002514835 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.371026+0000", 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9716999999999998e-05 2024-09-04T21:55:41.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1379 3.a e109/108 3:573a0ada:::rbd_data.11885236e45a.00000000000000e4:1d v 109'577, mlcod=0'0)", 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.57811921200000005, 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043781289999999997, 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:41.994 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:41.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366384+0000", 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1800000000000005e-07 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366392+0000", 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2439999999999993e-06 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366404+0000", 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2510000000000001e-05 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366416+0000", 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1361e-05 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368895+0000", 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024789529999999999 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368917+0000", 2024-09-04T21:55:41.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2303999999999999e-05 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370714+0000", 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017969100000000001 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370761+0000", 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7129000000000002e-05 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:41.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:143697 3.13 e109/108 3:ceb3a275:::rbd_object_map.1188388e16dc:head v 109'911, mlcod=0'0)", 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.371356+0000", 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.57314577799999999, 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003184493, 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:41.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371356+0000", 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371356+0000", 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371357+0000", 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1500000000000005e-07 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371363+0000", 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7969999999999997e-06 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371371+0000", 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3229999999999998e-06 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371380+0000", 2024-09-04T21:55:42.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7110000000000001e-06 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371757+0000", 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000376948 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371779+0000", 2024-09-04T21:55:42.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2532999999999999e-05 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374493+0000", 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027140670000000001 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374541+0000", 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7599000000000003e-05 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.003 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1380 3.a e109/108 3:54d284e2:::rbd_data.1188e6911c8f.0000000000000046:1e v 109'579, mlcod=107'575)", 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.57026971500000001, 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039209320000000002, 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374233+0000", 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-04T21:55:42.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374240+0000", 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6939999999999998e-06 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374251+0000", 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1644000000000001e-05 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374262+0000", 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0691e-05 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375369+0000", 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001106592 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375423+0000", 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4483e-05 2024-09-04T21:55:42.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378103+0000", 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002680151 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378153+0000", 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9947999999999997e-05 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2962 3.e e109/108 3:75446c9d:::rbd_data.1188af11ea53.000000000000003d:20 v 109'353, mlcod=0'0)", 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.379059+0000", 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56544322400000002, 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029576310000000001, 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379059+0000", 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379059+0000", 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379060+0000", 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3e-07 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379065+0000", 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4009999999999996e-06 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379075+0000", 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0084e-05 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379081+0000", 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.66e-06 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379705+0000", 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062451399999999995 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379724+0000", 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8573e-05 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381887+0000", 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021633289999999999 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.382016+0000", 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012934000000000001 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2963 3.e e109/108 3:757a8394:::rbd_data.1188af11ea53.000000000000007f:20 v 109'355, mlcod=0'0)", 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.379974+0000", 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56452819099999996, 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003607655, 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379974+0000", 2024-09-04T21:55:42.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379974+0000", 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379975+0000", 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0299999999999998e-07 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379980+0000", 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7150000000000003e-06 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379990+0000", 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6709999999999994e-06 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379997+0000", 2024-09-04T21:55:42.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4479999999999999e-06 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.380274+0000", 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027735699999999999 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.380298+0000", 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4134999999999998e-05 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383473+0000", 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003174839 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383582+0000", 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108687 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1381 3.a e109/108 3:501a596b:::rbd_data.1188af11ea53.000000000000012d:20 v 109'581, mlcod=109'577)", 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.381823+0000", 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56267939700000003, 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028750849999999999, 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381823+0000", 2024-09-04T21:55:42.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381823+0000", 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381824+0000", 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8999999999999995e-07 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381830+0000", 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3770000000000001e-06 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381842+0000", 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2160999999999999e-05 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.381851+0000", 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4630000000000001e-06 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383198+0000", 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013473739999999999 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383216+0000", 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8368000000000001e-05 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.384647+0000", 2024-09-04T21:55:42.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001430408 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.384698+0000", 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1044000000000001e-05 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2964 3.13 e109/108 3:ccb4316a:::rbd_data.1188af11ea53.000000000000002c:20 v 109'915, mlcod=109'911)", 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.383283+0000", 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56121935099999998, 2024-09-04T21:55:42.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028012610000000002, 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383283+0000", 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383283+0000", 2024-09-04T21:55:42.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383283+0000", 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2699999999999999e-07 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383289+0000", 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4770000000000002e-06 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383299+0000", 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7210000000000006e-06 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383306+0000", 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1670000000000002e-06 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383490+0000", 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000184062 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383546+0000", 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6542000000000002e-05 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386021+0000", 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002474778 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386084+0000", 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2886999999999996e-05 2024-09-04T21:55:42.027 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2965 3.13 e109/108 3:cc7f4b96:::rbd_data.1188af11ea53.000000000000001c:20 v 109'917, mlcod=109'911)", 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.56117912400000003, 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039308429999999998, 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3000000000000001e-07 2024-09-04T21:55:42.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383328+0000", 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6419999999999998e-06 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383337+0000", 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6270000000000006e-06 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383344+0000", 2024-09-04T21:55:42.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1280000000000002e-06 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385347+0000", 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020028759999999998 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385361+0000", 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4578e-05 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387203+0000", 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018416210000000001 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387254+0000", 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0940999999999999e-05 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1383 3.a e109/108 3:53a15561:::rbd_data.1188af11ea53.000000000000001b:20 v 109'585, mlcod=109'581)", 2024-09-04T21:55:42.032 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.55837820699999996, 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041799580000000001, 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2699999999999999e-07 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386129+0000", 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5700000000000003e-06 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386138+0000", 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9449999999999999e-06 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386144+0000", 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3749999999999999e-06 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386257+0000", 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112911 2024-09-04T21:55:42.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386305+0000", 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8087999999999998e-05 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390256+0000", 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003950968 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390304+0000", 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7574000000000002e-05 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2971 3.13 e109/108 3:cf71bd62:::rbd_data.1188af11ea53.000000000000010a:20 v 109'925, mlcod=109'919)", 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.393261+0000", 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.55124060100000005, 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032048710000000002, 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.037 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393261+0000", 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393261+0000", 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393262+0000", 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393268+0000", 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3449999999999996e-06 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393277+0000", 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1819999999999995e-06 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393283+0000", 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393450+0000", 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000166713 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393472+0000", 2024-09-04T21:55:42.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1934e-05 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.396419+0000", 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029470759999999999 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.396466+0000", 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7713000000000002e-05 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.041 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143860 3.1c 3:3f146d06:::rbd_object_map.118816ab2dec.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.592876+0000", 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.351625721, 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028707889999999999, 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:42.042 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143860 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.592876+0000", 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.592876+0000", 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.592879+0000", 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.593e-06 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.592890+0000", 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0896999999999999e-05 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.592921+0000", 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1335999999999999e-05 2024-09-04T21:55:42.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.592936+0000", 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4603000000000001e-05 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.593097+0000", 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000161609 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.593357+0000", 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025985100000000002 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.593697+0000", 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033965200000000002 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.595303+0000", 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016056149999999999 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.595586+0000", 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028397300000000003 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.595673+0000", 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6525000000000003e-05 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.595747+0000", 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4134999999999994e-05 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.047 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:42.048 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:42.048 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:42.049 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:42.047+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:42.050 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:42.072 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:42.072 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:42.099 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:42.099 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:42.125 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:42.126 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:42.150 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:42.143+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:42.152 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:42.152 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:42.173 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:42.173 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:42.179 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:42.179 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:42.188 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:42.251 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:42.247+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:42.352 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:42.347+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:42.364 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:42.364 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.2 to 1 (10000) 2024-09-04T21:55:42.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:42.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:55:42.399 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-04T21:55:42.453 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:42.447+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:42.520 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:42.520 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:42.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:42.554 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:42.547+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:42.655 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:42.651+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:42.658 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:42.658 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:42.673 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:42.756 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:42.751+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:42.828 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:42.828 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:42.842 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:42.890 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:42.887+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:42.957 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:42.951+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:42.960 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:42.960 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:42.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:43.062 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:43.059+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:43.141 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:43.141 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:43.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:43.198 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:43.195+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:43.260 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:43.255+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:43.320 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:43.320 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:43.335 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:43.361 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:43.355+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:43.380 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.3 to 1 (10000) 2024-09-04T21:55:43.388 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:55:43.412 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-04T21:55:43.502 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:43.499+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:43.506 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:43.506 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:43.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:43.563 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:43.555+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:43.664 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:43.659+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:43.686 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:43.686 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:43.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:43.786 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:43.783+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:43.850 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:43.850 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:43.862 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:43.906 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:43.903+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:43.967 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:43.963+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:44.019 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:44.019 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:44.032 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:44.074 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:44.071+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:44.163 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:44.163 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:44.168 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:44.163+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:44.177 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:44.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:44.269 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:44.263+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:44.341 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:44.341 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:44.342 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:44.343 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:44.356 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:44.356 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:44.357 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:44.370 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:44.367+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:44.372 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:44.372 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:44.381 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.4 to 1 (10000) 2024-09-04T21:55:44.391 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:44.391 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:44.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-04T21:55:44.411 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:44.411 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:44.430 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:44.430 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:44.462 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:44.462 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:44.470 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:44.463+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:44.483 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:44.483 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26348 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.089998816999994, 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057606602, 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.534 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26348 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441765+0000", 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441768+0000", 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6529999999999998e-06 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441780+0000", 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2054e-05 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441812+0000", 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1881999999999997e-05 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441827+0000", 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5461999999999999e-05 2024-09-04T21:55:44.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.441977+0000", 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014946800000000001 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.442230+0000", 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025264599999999998 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:44.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.451954+0000", 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0097241749999999998 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499009+0000", 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047055155000000001 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499139+0000", 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012970599999999999 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499237+0000", 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8706e-05 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.499372+0000", 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000134695 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:26394 3.1c 3:3edc4c8d:::rbd_data.118852763fbb.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.905457761999998, 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055547609999999997, 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 26394 2024-09-04T21:55:44.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626306+0000", 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626308+0000", 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0480000000000001e-06 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626318+0000", 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8600000000000005e-06 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626345+0000", 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6707e-05 2024-09-04T21:55:44.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626360+0000", 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4973999999999999e-05 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626473+0000", 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112731 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.626616+0000", 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014278399999999999 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.638326+0000", 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011710735999999999 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681529+0000", 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.043203076 2024-09-04T21:55:44.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681655+0000", 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125549 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681774+0000", 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119251 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:25.681854+0000", 2024-09-04T21:55:44.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9894000000000005e-05 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43351 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572162332999994, 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051338846, 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43351 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:44.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959602+0000", 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4499999999999997e-07 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959605+0000", 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959613+0000", 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8250000000000005e-06 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959618+0000", 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.417e-06 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010746+0000", 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051127924999999998 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010835+0000", 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9444000000000006e-05 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010941+0000", 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105312 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.551 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43353 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1269760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572138280999994, 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051544344999999998, 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43353 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959626+0000", 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3300000000000003e-07 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959629+0000", 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2560000000000001e-06 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959635+0000", 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0430000000000004e-06 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959640+0000", 2024-09-04T21:55:44.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0320000000000003e-06 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.010976+0000", 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051336248000000001 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011067+0000", 2024-09-04T21:55:44.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0826000000000001e-05 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011170+0000", 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103507 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43355 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572116139000002, 2024-09-04T21:55:44.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051760575000000003, 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43355 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959648+0000", 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7800000000000002e-07 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959651+0000", 2024-09-04T21:55:44.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959658+0000", 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5730000000000002e-06 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959663+0000", 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011205+0000", 2024-09-04T21:55:44.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051541489000000003 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011296+0000", 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1105000000000003e-05 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011409+0000", 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112927 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43358 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572093174000003, 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051974999000000001, 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43358 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.561 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959671+0000", 2024-09-04T21:55:44.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9999999999999998e-07 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959674+0000", 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.503e-06 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959682+0000", 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9500000000000001e-06 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959687+0000", 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1420000000000001e-06 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011451+0000", 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051763862000000001 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011538+0000", 2024-09-04T21:55:44.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7322000000000002e-05 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011646+0000", 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010772000000000001 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43361 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572069756999994, 2024-09-04T21:55:44.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052194391, 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43361 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:44.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959694+0000", 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959695+0000", 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6699999999999999e-07 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959697+0000", 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.418e-06 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959703+0000", 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9989999999999997e-06 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959708+0000", 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.904e-06 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011687+0000", 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051978506000000001 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011781+0000", 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4301000000000004e-05 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011889+0000", 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107796 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.569 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43362 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572048965999997, 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052922181999999998, 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.570 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43362 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959715+0000", 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959716+0000", 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3300000000000002e-07 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959718+0000", 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959725+0000", 2024-09-04T21:55:44.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4660000000000002e-06 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959730+0000", 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1019999999999996e-06 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.011924+0000", 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052193145000000003 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012015+0000", 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0969000000000003e-05 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012637+0000", 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062277700000000003 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43364 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1351680~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.574 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572026562999994, 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052915188000000002, 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43364 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959738+0000", 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3799999999999997e-07 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959740+0000", 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.306e-06 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959747+0000", 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9149999999999999e-06 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959754+0000", 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8000000000000001e-06 2024-09-04T21:55:44.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012425+0000", 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052670531 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012525+0000", 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000100581 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012653+0000", 2024-09-04T21:55:44.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012751700000000001 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43366 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.572002914999999, 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052927226000000001, 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.579 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43366 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959761+0000", 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959762+0000", 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0399999999999996e-07 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.959764+0000", 2024-09-04T21:55:44.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3949999999999999e-06 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960370+0000", 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060537999999999998 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960389+0000", 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9326000000000001e-05 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012680+0000", 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052291232 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012688+0000", 2024-09-04T21:55:44.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3890000000000005e-06 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43368 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.571344229999994, 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.052287419000000002, 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43368 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960420+0000", 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960422+0000", 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.84e-06 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960431+0000", 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.816e-06 2024-09-04T21:55:44.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960971+0000", 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000539965 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.960991+0000", 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0832999999999999e-05 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012703+0000", 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051711615000000002 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012707+0000", 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3499999999999999e-06 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43369 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:44.590 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570743523999994, 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051701824, 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43369 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961021+0000", 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961022+0000", 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6139999999999999e-06 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961030+0000", 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1489999999999994e-06 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961051+0000", 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961065+0000", 2024-09-04T21:55:44.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4163e-05 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012718+0000", 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051653533000000001 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012722+0000", 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0149999999999997e-06 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.594 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43371 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570678462000004, 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051652328999999997, 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.595 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43371 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961086+0000", 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961087+0000", 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2359999999999999e-06 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961106+0000", 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9097e-05 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961146+0000", 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0281999999999998e-05 2024-09-04T21:55:44.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961162+0000", 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5993000000000001e-05 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012734+0000", 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051571744000000003 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012738+0000", 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9770000000000002e-06 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43373 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570580754999995, 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051570802999999998, 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.599 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43373 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961183+0000", 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961185+0000", 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.423e-06 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961191+0000", 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9429999999999997e-06 2024-09-04T21:55:44.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961210+0000", 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8943000000000001e-05 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961223+0000", 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3506000000000001e-05 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012750+0000", 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051527008999999999 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012754+0000", 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9790000000000004e-06 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.603 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43375 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570521580000005, 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051526660000000002, 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43375 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961243+0000", 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961244+0000", 2024-09-04T21:55:44.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961249+0000", 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1699999999999996e-06 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961270+0000", 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0925999999999999e-05 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961283+0000", 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2986000000000001e-05 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012765+0000", 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0514825 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012769+0000", 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8079999999999998e-06 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43377 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570464345000005, 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051486632999999997, 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.608 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43377 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961300+0000", 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961301+0000", 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961306+0000", 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.69e-06 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961323+0000", 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6976000000000001e-05 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961336+0000", 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2913e-05 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012781+0000", 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051444996999999999 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012786+0000", 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9340000000000003e-06 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43379 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570411699000005, 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051454868000000001, 2024-09-04T21:55:44.612 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43379 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961352+0000", 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961354+0000", 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0729999999999999e-06 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961358+0000", 2024-09-04T21:55:44.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6099999999999999e-06 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961375+0000", 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7091999999999998e-05 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961390+0000", 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4803e-05 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012803+0000", 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051413343 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012807+0000", 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9469999999999996e-06 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43381 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570355270999997, 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051415832000000002, 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43381 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.617 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961409+0000", 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961410+0000", 2024-09-04T21:55:44.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1230000000000001e-06 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961415+0000", 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.977e-06 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961431+0000", 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6016e-05 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961444+0000", 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2816000000000001e-05 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012821+0000", 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051376809000000002 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012825+0000", 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0910000000000003e-06 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.620 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43383 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570302143000006, 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051378030999999998, 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.621 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43383 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961462+0000", 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961463+0000", 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1999999999999999e-06 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961473+0000", 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961535+0000", 2024-09-04T21:55:44.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2787999999999996e-05 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961555+0000", 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9646000000000001e-05 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012836+0000", 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051281032999999997 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012840+0000", 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9840000000000003e-06 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:43385 3.5 3.a3654ac5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.570174195999996, 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051265590999999999, 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43385 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961590+0000", 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961591+0000", 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961597+0000", 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961615+0000", 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7836e-05 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:39.961628+0000", 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2976000000000001e-05 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012851+0000", 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051223109000000003 2024-09-04T21:55:44.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:40.012856+0000", 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7870000000000001e-06 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:44.629 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:44.631 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:44.631 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:44.627+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:44.731 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:44.727+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:44.804 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:44.805 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:44.805 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:44.805 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:44.819 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:44.859 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:44.855+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:44.933 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:44.927+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:44.976 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:44.976 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:44.977 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:44.991 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:45.035 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:45.031+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.753465808999998, 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49175 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 475136~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320769384000002, 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021719833000000001, 2024-09-04T21:55:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49175 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793193+0000", 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4e-07 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793196+0000", 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793206+0000", 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4229999999999994e-06 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793213+0000", 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4370000000000001e-06 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814388+0000", 2024-09-04T21:55:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021174454999999998 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814604+0000", 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021587900000000001 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814912+0000", 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030848600000000001 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49290 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 40960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320738720999998, 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022433502000000001, 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49290 2024-09-04T21:55:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793224+0000", 2024-09-04T21:55:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6599999999999996e-07 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793227+0000", 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793240+0000", 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-05 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793252+0000", 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2843000000000001e-05 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815053+0000", 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021800657000000001 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815301+0000", 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024764399999999999 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815656+0000", 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035569200000000001 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49297 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 106496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320698028999999, 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022960876000000002, 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49297 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793268+0000", 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4460000000000001e-06 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793277+0000", 2024-09-04T21:55:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2299999999999997e-06 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793284+0000", 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9530000000000002e-06 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815785+0000", 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022500857999999999 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815996+0000", 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211065 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816224+0000", 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022851699999999999 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49303 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 172032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320668206999997, 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023578159000000001, 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49303 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793294+0000", 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793297+0000", 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9500000000000001e-06 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793306+0000", 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2140000000000002e-06 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793313+0000", 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816342+0000", 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023028835000000001 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816580+0000", 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023813 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816872+0000", 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029135000000000001 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49309 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 237568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320597245000002, 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024080300999999998, 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49309 2024-09-04T21:55:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793365+0000", 2024-09-04T21:55:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.054e-06 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793374+0000", 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793388+0000", 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3403e-05 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793396+0000", 2024-09-04T21:55:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8830000000000006e-06 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816958+0000", 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023561992 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817170+0000", 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021269799999999999 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817445+0000", 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027435799999999999 2024-09-04T21:55:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49326 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 303104~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320522322999999, 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024582155000000001, 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49326 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793440+0000", 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.003e-06 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793445+0000", 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6840000000000004e-06 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793464+0000", 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9085000000000001e-05 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793478+0000", 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3716999999999999e-05 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817523+0000", 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024045704000000001 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817738+0000", 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021439899999999999 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818021+0000", 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028356300000000002 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49332 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320470204000003, 2024-09-04T21:55:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025083616, 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49332 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793492+0000", 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793496+0000", 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793506+0000", 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9729999999999992e-06 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793516+0000", 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6269999999999995e-06 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818099+0000", 2024-09-04T21:55:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024583880999999998 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818307+0000", 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020786200000000001 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818575+0000", 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026768400000000002 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49338 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320436018000002, 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025617277000000001, 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49338 2024-09-04T21:55:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:45.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5600000000000005e-07 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793533+0000", 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1930000000000004e-06 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793542+0000", 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0669999999999998e-06 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793552+0000", 2024-09-04T21:55:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1290000000000002e-06 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818661+0000", 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025109056000000001 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818870+0000", 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000208929 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819143+0000", 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027324700000000001 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49344 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.320400032999999, 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026178261000000001, 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49344 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:45.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793565+0000", 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8930000000000001e-06 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793574+0000", 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793581+0000", 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8399999999999997e-06 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819234+0000", 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025653441999999999 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819463+0000", 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022879500000000001 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819740+0000", 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027665399999999998 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51043 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986862862000002, 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022470637000000002, 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51043 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127100+0000", 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127114+0000", 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4017e-05 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127128+0000", 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4005000000000001e-05 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127139+0000", 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1007e-05 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148542+0000", 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021403575000000001 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148727+0000", 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018442900000000001 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149569+0000", 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084262400000000004 2024-09-04T21:55:45.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51044 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 262144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986808256000003, 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023138729, 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51044 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127154+0000", 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127158+0000", 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.172e-06 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127194+0000", 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5222000000000003e-05 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127206+0000", 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2159e-05 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149626+0000", 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022420476000000002 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149779+0000", 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015296299999999999 2024-09-04T21:55:45.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150292+0000", 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051277400000000002 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51045 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 360448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986737708, 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023446379, 2024-09-04T21:55:45.172 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51045 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127225+0000", 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127230+0000", 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4110000000000004e-06 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127245+0000", 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4714e-05 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127256+0000", 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0787e-05 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150350+0000", 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023094001999999999 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150502+0000", 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151805 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150670+0000", 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016861 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51046 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.176 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986689380999998, 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023742831999999998, 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51046 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127273+0000", 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.23e-07 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127277+0000", 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9650000000000002e-06 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127291+0000", 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3549e-05 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127302+0000", 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1270000000000001e-05 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150727+0000", 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023424996999999999 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150882+0000", 2024-09-04T21:55:45.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015530200000000001 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151015+0000", 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013292600000000001 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51047 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 557056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986646247000003, 2024-09-04T21:55:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024031869000000001, 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51047 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127316+0000", 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5300000000000003e-07 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127320+0000", 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0749999999999999e-06 2024-09-04T21:55:45.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127334+0000", 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3587000000000001e-05 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127344+0000", 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0461e-05 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151070+0000", 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023725311999999998 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151219+0000", 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014918599999999999 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151347+0000", 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012839500000000001 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51048 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:45.185 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986603770000002, 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024346703000000001, 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51048 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127359+0000", 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.85e-07 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127375+0000", 2024-09-04T21:55:45.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6084000000000001e-05 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127389+0000", 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.412e-05 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127400+0000", 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151402+0000", 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024002787000000001 2024-09-04T21:55:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151573+0000", 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170941 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151705+0000", 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131161 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.189 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51049 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986534323999997, 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024613729000000001, 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51049 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127428+0000", 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6700000000000002e-07 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127433+0000", 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0000000000000004e-06 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127447+0000", 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4224e-05 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127459+0000", 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1112e-05 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151761+0000", 2024-09-04T21:55:45.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024302397999999999 2024-09-04T21:55:45.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151914+0000", 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015300399999999999 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152041+0000", 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127024 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.196 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51050 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986488021, 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024957087999999999, 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51050 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127479+0000", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7160000000000002e-06 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127494+0000", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4997000000000001e-05 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127506+0000", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.168e-05 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152104+0000", 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024597810000000001 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152273+0000", 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016888100000000001 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152431+0000", 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000158097 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.200 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51051 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.986440526999999, 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025255452000000001, 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51051 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.201 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127522+0000", 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:55:45.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127527+0000", 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6299999999999997e-06 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127541+0000", 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4225e-05 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127552+0000", 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0971999999999999e-05 2024-09-04T21:55:45.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152484+0000", 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024931803999999998 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152641+0000", 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015691699999999999 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152776+0000", 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013595400000000001 2024-09-04T21:55:45.204 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:106971 3.0 e82/38 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head v 82'257, mlcod=81'254)", 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.484146199000001, 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045768883000000003, 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629817+0000", 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9400000000000001e-06 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629834+0000", 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6246999999999999e-05 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629860+0000", 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6628999999999999e-05 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629873+0000", 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3216e-05 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629999+0000", 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012590799999999999 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.630064+0000", 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4546000000000002e-05 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675498+0000", 2024-09-04T21:55:45.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045433922000000002 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675584+0000", 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6475000000000001e-05 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.209 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:45.211 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:45.212 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:45.203+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:45.311 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:45.307+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:45.382 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:45.383 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:45.383 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:45.383 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:45.389 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.5 to 1 (10000) 2024-09-04T21:55:45.400 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:45.416 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:55:45.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-04T21:55:45.447 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:45.443+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:45.512 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:45.507+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:45.533 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:45.534 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:45.549 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:45.635 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:45.631+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:45.714 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:45.707+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:45.716 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 75.098038196000005, 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:45.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:45.721 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 75.087032953000005, 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:45.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:45.726 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543326203000007, 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:45.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:45.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543285517000001, 2024-09-04T21:55:45.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:45.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543252117999998, 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.736 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:45.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:45.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543208847000002, 2024-09-04T21:55:45.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:45.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:45.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.744 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543164165999997, 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.745 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:45.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543103009999996, 2024-09-04T21:55:45.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:45.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.753 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543072968999994, 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.754 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:45.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:45.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.543028198000002, 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.758 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:45.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.542979905999999, 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.542944261000002, 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.767 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:45.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.542695045000002, 2024-09-04T21:55:45.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:45.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:45.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.542656175999994, 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:45.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:45.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.537509431000004, 2024-09-04T21:55:45.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:45.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:45.784 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.537466171999995, 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.785 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:45.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:45.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.537422301000007, 2024-09-04T21:55:45.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:45.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.537386022999996, 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:45.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:45.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.537359917000003, 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.798 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:45.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.802 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143697 3.13 3:ceb3a275:::rbd_object_map.1188388e16dc:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3426344779999999, 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.004254991, 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143697 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370766+0000", 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7429999999999998e-06 2024-09-04T21:55:45.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370776+0000", 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8549999999999997e-06 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370795+0000", 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9599e-05 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370807+0000", 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1652e-05 2024-09-04T21:55:45.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370896+0000", 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104000000000006e-05 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:45.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370911+0000", 2024-09-04T21:55:45.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4986e-05 2024-09-04T21:55:45.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:45.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370490+0000", 2024-09-04T21:55:45.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99957913600000003 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370701+0000", 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021104399999999999 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371058+0000", 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035635700000000002 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:45.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374510+0000", 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034528860000000001 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374898+0000", 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038715800000000002 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374971+0000", 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3881999999999995e-05 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:45.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375018+0000", 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6588999999999997e-05 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:45.811 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:45.812 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:45.815 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:45.815+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:45.916 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:45.911+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:45.983 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:45.983 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:45.983 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:45.983 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:45.997 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:46.043 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:46.039+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:46.117 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:46.111+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:46.123 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:46.124 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:46.124 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:46.124 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:46.124 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:46.138 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:46.223 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:46.219+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23, version: 95'461, data_included: [0~262144,327680~65536,507904~15827,3260416~65536,3407872~32768,3555328~65536,3670016~32768,3850240~32768], data_size: 572883, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23@95'461, size: 3883008, copy_subset: [(0, 3883008)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3883008, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.491600869, 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060935900000000003, 2024-09-04T21:55:46.306 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811327+0000", 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3629999999999999e-06 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811906+0000", 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057844000000000005 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811940+0000", 2024-09-04T21:55:46.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4096e-05 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811967+0000", 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7202000000000001e-05 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812156+0000", 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189073 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812200+0000", 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3748000000000002e-05 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.817419+0000", 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052196680000000002 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23, version: 95'467, data_included: [2883584~32768,3080192~16384,3145728~65536,3342336~49152,3440640~32768,3620864~49152], data_size: 245760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23@95'467, size: 3670016, copy_subset: [(0, 3670016)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3670016, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.4736075749999999, 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044989239999999996, 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829321+0000", 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5999999999999999e-06 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829583+0000", 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026244899999999999 2024-09-04T21:55:46.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829622+0000", 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8359000000000003e-05 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829679+0000", 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7102999999999999e-05 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829878+0000", 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019976299999999999 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829903+0000", 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4604999999999999e-05 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.833818+0000", 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039150449999999998 2024-09-04T21:55:46.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2959 3.f e108/105 3:f7c4ed88:::rbd_data.1188388e16dc.0000000000000001:1c v 108'438, mlcod=107'435)", 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.9382788499999997, 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055309019999999999, 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364649+0000", 2024-09-04T21:55:46.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364676+0000", 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7135e-05 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364688+0000", 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2082000000000001e-05 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364700+0000", 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1885999999999999e-05 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364799+0000", 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9593999999999995e-05 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364831+0000", 2024-09-04T21:55:46.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1628000000000003e-05 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370108+0000", 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052768629999999997 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370179+0000", 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1046999999999999e-05 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1379 3.a e109/108 3:573a0ada:::rbd_data.11885236e45a.00000000000000e4:1d v 109'577, mlcod=0'0)", 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9365438089999998, 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043781289999999997, 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366384+0000", 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1800000000000005e-07 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366392+0000", 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2439999999999993e-06 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366404+0000", 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2510000000000001e-05 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366416+0000", 2024-09-04T21:55:46.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1361e-05 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368895+0000", 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024789529999999999 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368917+0000", 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2303999999999999e-05 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370714+0000", 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017969100000000001 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370761+0000", 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7129000000000002e-05 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:143697 3.13 e109/108 3:ceb3a275:::rbd_object_map.1188388e16dc:head v 109'911, mlcod=0'0)", 2024-09-04T21:55:46.324 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.371356+0000", 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9315703749999997, 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003184493, 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371356+0000", 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371356+0000", 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371357+0000", 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1500000000000005e-07 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371363+0000", 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7969999999999997e-06 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371371+0000", 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3229999999999998e-06 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371380+0000", 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7110000000000001e-06 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371757+0000", 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000376948 2024-09-04T21:55:46.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371779+0000", 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2532999999999999e-05 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374493+0000", 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027140670000000001 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374541+0000", 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7599000000000003e-05 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1380 3.a e109/108 3:54d284e2:::rbd_data.1188e6911c8f.0000000000000046:1e v 109'579, mlcod=107'575)", 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9286943120000002, 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039209320000000002, 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.329 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374233+0000", 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374240+0000", 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6939999999999998e-06 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374251+0000", 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1644000000000001e-05 2024-09-04T21:55:46.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374262+0000", 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0691e-05 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375369+0000", 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001106592 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375423+0000", 2024-09-04T21:55:46.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4483e-05 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378103+0000", 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002680151 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378153+0000", 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9947999999999997e-05 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2963 3.e e109/108 3:757a8394:::rbd_data.1188af11ea53.000000000000007f:20 v 109'355, mlcod=0'0)", 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.379974+0000", 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9229527879999999, 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003607655, 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379974+0000", 2024-09-04T21:55:46.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379974+0000", 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379975+0000", 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0299999999999998e-07 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379980+0000", 2024-09-04T21:55:46.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7150000000000003e-06 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379990+0000", 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6709999999999994e-06 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.379997+0000", 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4479999999999999e-06 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.380274+0000", 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027735699999999999 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.380298+0000", 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4134999999999998e-05 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383473+0000", 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003174839 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383582+0000", 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108687 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2965 3.13 e109/108 3:cc7f4b96:::rbd_data.1188af11ea53.000000000000001c:20 v 109'917, mlcod=109'911)", 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9196037209999997, 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039308429999999998, 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.338 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:46.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3000000000000001e-07 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383328+0000", 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6419999999999998e-06 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383337+0000", 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6270000000000006e-06 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383344+0000", 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1280000000000002e-06 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385347+0000", 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020028759999999998 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385361+0000", 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4578e-05 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387203+0000", 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018416210000000001 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387254+0000", 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0940999999999999e-05 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1383 3.a e109/108 3:53a15561:::rbd_data.1188af11ea53.000000000000001b:20 v 109'585, mlcod=109'581)", 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:46.342 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9168028039999996, 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041799580000000001, 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2699999999999999e-07 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386129+0000", 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5700000000000003e-06 2024-09-04T21:55:46.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386138+0000", 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9449999999999999e-06 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386144+0000", 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3749999999999999e-06 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386257+0000", 2024-09-04T21:55:46.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112911 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386305+0000", 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8087999999999998e-05 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390256+0000", 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003950968 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390304+0000", 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7574000000000002e-05 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2971 3.13 e109/108 3:cf71bd62:::rbd_data.1188af11ea53.000000000000010a:20 v 109'925, mlcod=109'919)", 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.393261+0000", 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.9096651979999999, 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032048710000000002, 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393261+0000", 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393261+0000", 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393262+0000", 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:46.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393268+0000", 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3449999999999996e-06 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393277+0000", 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1819999999999995e-06 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393283+0000", 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393450+0000", 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000166713 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.393472+0000", 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1934e-05 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.396419+0000", 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0029470759999999999 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.396466+0000", 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7713000000000002e-05 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144273 3.9 3:92e794e6:::rbd_data.1188f756fcf9.0000000000000000:head [delete] snapc 19=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:44.465105+0000", 2024-09-04T21:55:46.351 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.8378215500000001, 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003010984, 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144273 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465105+0000", 2024-09-04T21:55:46.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465105+0000", 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465106+0000", 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0640000000000001e-06 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465111+0000", 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9479999999999999e-06 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465125+0000", 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3397999999999999e-05 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465132+0000", 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1729999999999998e-06 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465287+0000", 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015488899999999999 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.465819+0000", 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053193200000000004 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.466149+0000", 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000330716 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.467471+0000", 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013214139999999999 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.467944+0000", 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047292800000000001 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468034+0000", 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0451999999999994e-05 2024-09-04T21:55:46.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468116+0000", 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2070000000000005e-05 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144283 3.1 3:86d0b0c6:::rbd_data.1188f756fcf9.0000000000000060:head [delete] snapc 19=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:44.468285+0000", 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.8346412249999999, 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003670038, 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.357 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144283 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468285+0000", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468285+0000", 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468287+0000", 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.669e-06 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468294+0000", 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.973e-06 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468314+0000", 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0373e-05 2024-09-04T21:55:46.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468326+0000", 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1104000000000001e-05 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.468437+0000", 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000111104 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.469052+0000", 2024-09-04T21:55:46.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061554199999999996 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.469267+0000", 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021495299999999999 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.471298+0000", 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020303510000000001 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:46.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.471867+0000", 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00056952499999999998 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.471914+0000", 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6482e-05 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:44.471955+0000", 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1962000000000002e-05 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3021 3.11 e113/105 3:8825ef77:::rbd_data.1188e5a3a295.0000000000000034:18 v 113'516, mlcod=106'512)", 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.397817+0000", 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.90511005499999997, 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035935400000000001, 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.363 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397817+0000", 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397817+0000", 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397818+0000", 2024-09-04T21:55:46.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0070000000000001e-06 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397829+0000", 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1e-05 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397857+0000", 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8776e-05 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397879+0000", 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1659e-05 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398041+0000", 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016217 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398106+0000", 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4825000000000004e-05 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401369+0000", 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032632130000000001 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401410+0000", 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0890000000000003e-05 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3023 3.11 e113/105 3:8c151454:::rbd_data.1188e5a3a295.0000000000000009:18 v 113'518, mlcod=106'512)", 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.90499213000000001, 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038753149999999998, 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4e-07 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397954+0000", 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8236e-05 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397974+0000", 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0528999999999999e-05 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397989+0000", 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4239999999999999e-05 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.399110+0000", 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00112105 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.399137+0000", 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7208000000000001e-05 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401756+0000", 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026188549999999998 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401810+0000", 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4256999999999999e-05 2024-09-04T21:55:46.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1400 3.a e113/108 3:568fad06:::rbd_data.1188e5a3a295.0000000000000006:18 v 113'597, mlcod=110'594)", 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.90478370200000002, 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037033109999999999, 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398144+0000", 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5300000000000004e-07 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398151+0000", 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6340000000000001e-06 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398192+0000", 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0862000000000003e-05 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398207+0000", 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.505e-05 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398405+0000", 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000197612 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398485+0000", 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0644999999999998e-05 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401799+0000", 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033140130000000002 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401846+0000", 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6842000000000003e-05 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3025 3.f e113/105 3:f69cda90:::rbd_data.1188e5a3a295.0000000000000004:18 v 113'446, mlcod=111'443)", 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.89873861399999999, 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070421360000000001, 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.376 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404189+0000", 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404195+0000", 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3149999999999997e-06 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404204+0000", 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0240000000000003e-06 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404213+0000", 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5040000000000008e-06 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404315+0000", 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010159599999999999 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404344+0000", 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9175000000000001e-05 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411189+0000", 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006844854 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411230+0000", 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1137999999999998e-05 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.3.0:4041 3.19 e113/105 3:98fe2de1:::rbd_data.1188e5a3a295.000000000000002a:18 v 113'1164, mlcod=110'1161)", 2024-09-04T21:55:46.380 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.89866763800000005, 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007607361, 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404260+0000", 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404265+0000", 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0119999999999996e-06 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404274+0000", 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.764e-06 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404282+0000", 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1169999999999998e-06 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404538+0000", 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025666600000000001 2024-09-04T21:55:46.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404564+0000", 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5415999999999999e-05 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411833+0000", 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0072691520000000001 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411866+0000", 2024-09-04T21:55:46.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3643000000000003e-05 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3026 3.f e113/105 3:f7074c4a:::rbd_data.1188e5a3a295.0000000000000024:18 v 113'448, mlcod=111'443)", 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.89827216200000004, 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073657970000000003, 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.385 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:46.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4799999999999997e-07 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404663+0000", 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404675+0000", 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2108e-05 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404683+0000", 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2719999999999997e-06 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405007+0000", 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032386699999999998 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405035+0000", 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7795000000000001e-05 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411899+0000", 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068636569999999996 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.412020+0000", 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121777 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144455 3.14 3:2faad357:::rbd_data.1188e5a3a295.0000000000000002:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-04T21:55:46.389 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.490426+0000", 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.81250085100000002, 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032269849999999999, 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144455 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490426+0000", 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490426+0000", 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490426+0000", 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7800000000000001e-07 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490429+0000", 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0450000000000001e-06 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490437+0000", 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6660000000000008e-06 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490443+0000", 2024-09-04T21:55:46.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9200000000000001e-06 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.491504+0000", 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010606680000000001 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.491584+0000", 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0632999999999999e-05 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:46.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.491678+0000", 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3523000000000006e-05 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.492937+0000", 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001259624 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493585+0000", 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000647413 2024-09-04T21:55:46.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493617+0000", 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2557e-05 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493653+0000", 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5358000000000003e-05 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144458 3.9 3:93d277ee:::rbd_data.1188e5a3a295.0000000000000005:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.490475+0000", 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.81245120599999998, 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0032117489999999999, 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144458 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.396 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490475+0000", 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490475+0000", 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490476+0000", 2024-09-04T21:55:46.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490479+0000", 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.88e-06 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490487+0000", 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5700000000000004e-06 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:46.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.490493+0000", 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1630000000000001e-06 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.491290+0000", 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079770899999999996 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.491369+0000", 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8991000000000003e-05 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.491493+0000", 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012362500000000001 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493116+0000", 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001622617 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493620+0000", 2024-09-04T21:55:46.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050474899999999997 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493659+0000", 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8470999999999998e-05 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.493687+0000", 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8371999999999999e-05 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.402 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:46.402 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:46.402 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:46.402 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:55:46.403 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.6 to 1 (10000) 2024-09-04T21:55:46.403 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:46.399+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:46.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:46.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-04T21:55:46.501 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:46.522 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:46.519+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:46.531 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:46.531 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:46.531 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:46.531 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:46.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:46.638 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:46.635+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:46.668 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:46.668 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:46.686 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:46.686 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:46.702 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:46.703 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:46.705 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:46.699+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:46.710 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:46.710 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:46.719 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:46.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:46.740 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:46.741 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:46.761 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:46.761 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:46.782 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:46.783 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:46.803 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:46.803 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:46.806 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:46.799+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:46.852 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:46.852 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:46.865 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:46.907 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:46.903+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:47.008 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:47.003+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:47.025 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.025 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:47.109 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:47.103+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:47.204 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.204 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.210 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:47.207+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:47.220 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:47.311 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:47.307+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:47.340 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.340 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:47.410 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.7 to 1 (10000) 2024-09-04T21:55:47.411 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:47.407+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:47.435 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-04T21:55:47.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:55:47.488 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.488 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:47.513 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:47.507+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:47.613 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:47.607+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:47.634 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.634 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.650 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:47.714 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:47.711+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:47.749 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:55:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:55:47.767 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-04T21:55:47.412320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630748243,"num_objects":972,"num_object_clones":224,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":369,"num_read":186304,"num_read_kb":668760,"num_write":19689,"num_write_kb":315755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1232,"num_bytes_recovered":57788192,"num_keys_recovered":237,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15895,"ondisk_log_size":15895,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":287892,"kb_used_data":36652,"kb_used_omap":43,"kb_used_meta":251028,"kb_avail":565943148,"statfs":{"total":579820584960,"available":579525783552,"internally_reserved":0,"allocated":37531648,"data_stored":264853763,"data_compressed":13616459,"data_compressed_allocated":23924736,"data_compressed_original":257598516,"omap_allocated":44792,"internal_metadata":257052936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-155716629,"num_objects":-157,"num_object_clones":-60,"num_object_copies":-314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-157,"num_whiteouts":-46,"num_read":696,"num_read_kb":604,"num_write":158,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":124562,"num_keys_recovered":18,"num_objects_omap":-3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030179"},"pg_stats":[{"pgid":"3.1f","version":"110'730","reported_seq":2662,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.399885+0000","last_change":"2024-09-04T21:55:42.400305+0000","last_active":"2024-09-04T21:55:42.399885+0000","last_peered":"2024-09-04T21:55:42.399885+0000","last_clean":"2024-09-04T21:55:42.399885+0000","last_became_active":"2024-09-04T21:54:51.350505+0000","last_became_peered":"2024-09-04T21:54:51.350505+0000","last_unstale":"2024-09-04T21:55:42.399885+0000","last_undegraded":"2024-09-04T21:55:42.399885+0000","last_fullsized":"2024-09-04T21:55:42.399885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:42:08.481832+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.018428050000000001,"stat_sum":{"num_bytes":15466997,"num_objects":21,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":12,"num_read":4923,"num_read_kb":7638,"num_write":1065,"num_write_kb":4569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":330234,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.1e","version":"110'355","reported_seq":3289,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.465147+0000","last_change":"2024-09-04T21:55:42.399425+0000","last_active":"2024-09-04T21:55:42.465147+0000","last_peered":"2024-09-04T21:55:42.465147+0000","last_clean":"2024-09-04T21:55:42.465147+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:55:42.465147+0000","last_undegraded":"2024-09-04T21:55:42.465147+0000","last_fullsized":"2024-09-04T21:55:42.465147+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:04:02.685368+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.025135099000000001,"stat_sum":{"num_bytes":15930092,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":13,"num_read":3609,"num_read_kb":13267,"num_write":334,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.1d","version":"112'508","reported_seq":18817,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.401624+0000","last_change":"2024-09-04T21:55:46.402623+0000","last_active":"2024-09-04T21:55:46.401624+0000","last_peered":"2024-09-04T21:55:46.401624+0000","last_clean":"2024-09-04T21:55:46.401624+0000","last_became_active":"2024-09-04T21:55:38.745373+0000","last_became_peered":"2024-09-04T21:55:38.745373+0000","last_unstale":"2024-09-04T21:55:46.401624+0000","last_undegraded":"2024-09-04T21:55:46.401624+0000","last_fullsized":"2024-09-04T21:55:46.401624+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:05:21.791227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050457210000000004,"stat_sum":{"num_bytes":33053972,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":8,"num_read":19674,"num_read_kb":23021,"num_write":691,"num_write_kb":9166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":911480,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.1c","version":"114'354","reported_seq":2169,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.550746+0000","last_change":"2024-09-04T21:55:46.404207+0000","last_active":"2024-09-04T21:55:46.550746+0000","last_peered":"2024-09-04T21:55:46.550746+0000","last_clean":"2024-09-04T21:55:46.550746+0000","last_became_active":"2024-09-04T21:55:38.745462+0000","last_became_peered":"2024-09-04T21:55:38.745462+0000","last_unstale":"2024-09-04T21:55:46.550746+0000","last_undegraded":"2024-09-04T21:55:46.550746+0000","last_fullsized":"2024-09-04T21:55:46.550746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:37:38.424780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006521186,"stat_sum":{"num_bytes":18878302,"num_objects":26,"num_object_clones":7,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":12,"num_read":2087,"num_read_kb":21010,"num_write":343,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.1b","version":"113'272","reported_seq":1491,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.400294+0000","last_change":"2024-09-04T21:55:46.401369+0000","last_active":"2024-09-04T21:55:46.400294+0000","last_peered":"2024-09-04T21:55:46.400294+0000","last_clean":"2024-09-04T21:55:46.400294+0000","last_became_active":"2024-09-04T21:55:38.763671+0000","last_became_peered":"2024-09-04T21:55:38.763671+0000","last_unstale":"2024-09-04T21:55:46.400294+0000","last_undegraded":"2024-09-04T21:55:46.400294+0000","last_fullsized":"2024-09-04T21:55:46.400294+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T21:58:18.921704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046880020000000001,"stat_sum":{"num_bytes":5396272,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":11,"num_read":2251,"num_read_kb":17532,"num_write":263,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2186006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.1a","version":"114'907","reported_seq":16937,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.630386+0000","last_change":"2024-09-04T21:55:46.398869+0000","last_active":"2024-09-04T21:55:46.630386+0000","last_peered":"2024-09-04T21:55:46.630386+0000","last_clean":"2024-09-04T21:55:46.630386+0000","last_became_active":"2024-09-04T21:55:41.365854+0000","last_became_peered":"2024-09-04T21:55:41.365854+0000","last_unstale":"2024-09-04T21:55:46.630386+0000","last_undegraded":"2024-09-04T21:55:46.630386+0000","last_fullsized":"2024-09-04T21:55:46.630386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":907,"log_dups_size":0,"ondisk_log_size":907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:59:36.861119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002204396,"stat_sum":{"num_bytes":28526938,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":14,"num_read":19752,"num_read_kb":56989,"num_write":1186,"num_write_kb":17244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1484312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.19","version":"110'1161","reported_seq":2921,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.533455+0000","last_change":"2024-09-04T21:55:42.415259+0000","last_active":"2024-09-04T21:55:42.533455+0000","last_peered":"2024-09-04T21:55:42.533455+0000","last_clean":"2024-09-04T21:55:42.533455+0000","last_became_active":"2024-09-04T21:55:38.757506+0000","last_became_peered":"2024-09-04T21:55:38.757506+0000","last_unstale":"2024-09-04T21:55:42.533455+0000","last_undegraded":"2024-09-04T21:55:42.533455+0000","last_fullsized":"2024-09-04T21:55:42.533455+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1161,"log_dups_size":0,"ondisk_log_size":1161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:25:33.800130+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.040984010000000001,"stat_sum":{"num_bytes":14223921,"num_objects":26,"num_object_clones":5,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":6610,"num_read_kb":17001,"num_write":1856,"num_write_kb":6539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":37560,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.18","version":"110'339","reported_seq":3915,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.385503+0000","last_change":"2024-09-04T21:55:42.387167+0000","last_active":"2024-09-04T21:55:42.385503+0000","last_peered":"2024-09-04T21:55:42.385503+0000","last_clean":"2024-09-04T21:55:42.385503+0000","last_became_active":"2024-09-04T21:54:51.349573+0000","last_became_peered":"2024-09-04T21:54:51.349573+0000","last_unstale":"2024-09-04T21:55:42.385503+0000","last_undegraded":"2024-09-04T21:55:42.385503+0000","last_fullsized":"2024-09-04T21:55:42.385503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:59:13.025001+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.013391735,"stat_sum":{"num_bytes":22865144,"num_objects":24,"num_object_clones":6,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":9,"num_read":4669,"num_read_kb":25777,"num_write":348,"num_write_kb":8113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3632211,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.17","version":"110'840","reported_seq":2700,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.579120+0000","last_change":"2024-09-04T21:55:42.416831+0000","last_active":"2024-09-04T21:55:42.579120+0000","last_peered":"2024-09-04T21:55:42.579120+0000","last_clean":"2024-09-04T21:55:42.579120+0000","last_became_active":"2024-09-04T21:54:51.352376+0000","last_became_peered":"2024-09-04T21:54:51.352376+0000","last_unstale":"2024-09-04T21:55:42.579120+0000","last_undegraded":"2024-09-04T21:55:42.579120+0000","last_fullsized":"2024-09-04T21:55:42.579120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:20:10.043365+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.042177032000000003,"stat_sum":{"num_bytes":31371140,"num_objects":35,"num_object_clones":2,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":14,"num_read":5136,"num_read_kb":21920,"num_write":1211,"num_write_kb":7086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3732324,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.16","version":"112'340","reported_seq":2511,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391858+0000","last_change":"2024-09-04T21:55:44.391858+0000","last_active":"2024-09-04T21:55:44.391858+0000","last_peered":"2024-09-04T21:55:44.391858+0000","last_clean":"2024-09-04T21:55:44.391858+0000","last_became_active":"2024-09-04T21:54:42.679389+0000","last_became_peered":"2024-09-04T21:54:42.679389+0000","last_unstale":"2024-09-04T21:55:44.391858+0000","last_undegraded":"2024-09-04T21:55:44.391858+0000","last_fullsized":"2024-09-04T21:55:44.391858+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:37:42.001276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010147750000000001,"stat_sum":{"num_bytes":13184683,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":8,"num_read":2897,"num_read_kb":16809,"num_write":322,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.15","version":"110'362","reported_seq":2937,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391127+0000","last_change":"2024-09-04T21:55:44.391562+0000","last_active":"2024-09-04T21:55:44.391127+0000","last_peered":"2024-09-04T21:55:44.391127+0000","last_clean":"2024-09-04T21:55:44.391127+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:55:44.391127+0000","last_undegraded":"2024-09-04T21:55:44.391127+0000","last_fullsized":"2024-09-04T21:55:44.391127+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:27:03.130370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076071899999999998,"stat_sum":{"num_bytes":18646762,"num_objects":43,"num_object_clones":10,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":22,"num_read":3202,"num_read_kb":27791,"num_write":383,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.14","version":"114'316","reported_seq":1850,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.400172+0000","last_change":"2024-09-04T21:55:46.400172+0000","last_active":"2024-09-04T21:55:46.400172+0000","last_peered":"2024-09-04T21:55:46.400172+0000","last_clean":"2024-09-04T21:55:46.400172+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:55:46.400172+0000","last_undegraded":"2024-09-04T21:55:46.400172+0000","last_fullsized":"2024-09-04T21:55:46.400172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:28:21.405856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033055609999999998,"stat_sum":{"num_bytes":4838177,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":7,"num_read":1562,"num_read_kb":6375,"num_write":344,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.7","version":"0'0","reported_seq":208,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.479511+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:55:43.479511+0000","last_peered":"2024-09-04T21:55:43.479511+0000","last_clean":"2024-09-04T21:55:43.479511+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:55:43.479511+0000","last_undegraded":"2024-09-04T21:55:43.479511+0000","last_fullsized":"2024-09-04T21:55:43.479511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"114'412","reported_seq":2234,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.605364+0000","last_change":"2024-09-04T21:55:46.402588+0000","last_active":"2024-09-04T21:55:46.605364+0000","last_peered":"2024-09-04T21:55:46.605364+0000","last_clean":"2024-09-04T21:55:46.605364+0000","last_became_active":"2024-09-04T21:55:41.369871+0000","last_became_peered":"2024-09-04T21:55:41.369871+0000","last_unstale":"2024-09-04T21:55:46.605364+0000","last_undegraded":"2024-09-04T21:55:46.605364+0000","last_fullsized":"2024-09-04T21:55:46.605364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:28:27.105299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051780849999999998,"stat_sum":{"num_bytes":4136554,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":3134,"num_read_kb":5309,"num_write":370,"num_write_kb":7369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":946773,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.6","version":"0'0","reported_seq":170,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.479625+0000","last_change":"2024-09-04T21:55:38.759143+0000","last_active":"2024-09-04T21:55:43.479625+0000","last_peered":"2024-09-04T21:55:43.479625+0000","last_clean":"2024-09-04T21:55:43.479625+0000","last_became_active":"2024-09-04T21:55:38.758298+0000","last_became_peered":"2024-09-04T21:55:38.758298+0000","last_unstale":"2024-09-04T21:55:43.479625+0000","last_undegraded":"2024-09-04T21:55:43.479625+0000","last_fullsized":"2024-09-04T21:55:43.479625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:32:19.422630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"110'296","reported_seq":3196,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.419249+0000","last_change":"2024-09-04T21:55:42.419566+0000","last_active":"2024-09-04T21:55:42.419249+0000","last_peered":"2024-09-04T21:55:42.419249+0000","last_clean":"2024-09-04T21:55:42.419249+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:55:42.419249+0000","last_undegraded":"2024-09-04T21:55:42.419249+0000","last_fullsized":"2024-09-04T21:55:42.419249+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:32:58.519392+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.044636157000000003,"stat_sum":{"num_bytes":10387018,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":10,"num_read":3238,"num_read_kb":21928,"num_write":326,"num_write_kb":14374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391031+0000","last_change":"2024-09-04T21:55:38.759967+0000","last_active":"2024-09-04T21:55:44.391031+0000","last_peered":"2024-09-04T21:55:44.391031+0000","last_clean":"2024-09-04T21:55:44.391031+0000","last_became_active":"2024-09-04T21:55:38.759102+0000","last_became_peered":"2024-09-04T21:55:38.759102+0000","last_unstale":"2024-09-04T21:55:44.391031+0000","last_undegraded":"2024-09-04T21:55:44.391031+0000","last_fullsized":"2024-09-04T21:55:44.391031+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:21:11.118581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"110'292","reported_seq":1619,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.498765+0000","last_change":"2024-09-04T21:55:42.389843+0000","last_active":"2024-09-04T21:55:42.498765+0000","last_peered":"2024-09-04T21:55:42.498765+0000","last_clean":"2024-09-04T21:55:42.498765+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:55:42.498765+0000","last_undegraded":"2024-09-04T21:55:42.498765+0000","last_fullsized":"2024-09-04T21:55:42.498765+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:31:27.322233+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0087944479999999998,"stat_sum":{"num_bytes":6730531,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":9,"num_read":1744,"num_read_kb":7801,"num_write":260,"num_write_kb":3585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.4","version":"0'0","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.390625+0000","last_change":"2024-09-04T21:54:48.372640+0000","last_active":"2024-09-04T21:55:44.390625+0000","last_peered":"2024-09-04T21:55:44.390625+0000","last_clean":"2024-09-04T21:55:44.390625+0000","last_became_active":"2024-09-04T21:54:48.372106+0000","last_became_peered":"2024-09-04T21:54:48.372106+0000","last_unstale":"2024-09-04T21:55:44.390625+0000","last_undegraded":"2024-09-04T21:55:44.390625+0000","last_fullsized":"2024-09-04T21:55:44.390625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:18:59.210273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"110'478","reported_seq":11459,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.409941+0000","last_change":"2024-09-04T21:55:42.410423+0000","last_active":"2024-09-04T21:55:42.409941+0000","last_peered":"2024-09-04T21:55:42.409941+0000","last_clean":"2024-09-04T21:55:42.409941+0000","last_became_active":"2024-09-04T21:54:42.679683+0000","last_became_peered":"2024-09-04T21:54:42.679683+0000","last_unstale":"2024-09-04T21:55:42.409941+0000","last_undegraded":"2024-09-04T21:55:42.409941+0000","last_fullsized":"2024-09-04T21:55:42.409941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:50:17.531826+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.035823607,"stat_sum":{"num_bytes":28978471,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":10,"num_read":12262,"num_read_kb":24996,"num_write":542,"num_write_kb":14132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.3","version":"114'404","reported_seq":3555,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.577396+0000","last_change":"2024-09-04T21:55:46.404031+0000","last_active":"2024-09-04T21:55:46.577396+0000","last_peered":"2024-09-04T21:55:46.577396+0000","last_clean":"2024-09-04T21:55:46.577396+0000","last_became_active":"2024-09-04T21:55:41.371293+0000","last_became_peered":"2024-09-04T21:55:41.371293+0000","last_unstale":"2024-09-04T21:55:46.577396+0000","last_undegraded":"2024-09-04T21:55:46.577396+0000","last_fullsized":"2024-09-04T21:55:46.577396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:16:32.630273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061329899999999996,"stat_sum":{"num_bytes":13250427,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":3237,"num_read_kb":16214,"num_write":460,"num_write_kb":12348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":2080930,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.2","version":"21'2","reported_seq":161,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.479355+0000","last_change":"2024-09-04T21:54:42.675259+0000","last_active":"2024-09-04T21:55:43.479355+0000","last_peered":"2024-09-04T21:55:43.479355+0000","last_clean":"2024-09-04T21:55:43.479355+0000","last_became_active":"2024-09-04T21:54:42.674803+0000","last_became_peered":"2024-09-04T21:54:42.674803+0000","last_unstale":"2024-09-04T21:55:43.479355+0000","last_undegraded":"2024-09-04T21:55:43.479355+0000","last_fullsized":"2024-09-04T21:55:43.479355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:09:21.260817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"110'321","reported_seq":2902,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.394233+0000","last_change":"2024-09-04T21:55:42.394233+0000","last_active":"2024-09-04T21:55:42.394233+0000","last_peered":"2024-09-04T21:55:42.394233+0000","last_clean":"2024-09-04T21:55:42.394233+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:55:42.394233+0000","last_undegraded":"2024-09-04T21:55:42.394233+0000","last_fullsized":"2024-09-04T21:55:42.394233+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:44:44.010352+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.013247919,"stat_sum":{"num_bytes":14463992,"num_objects":40,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":15,"num_read":3396,"num_read_kb":53595,"num_write":554,"num_write_kb":21343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.1","version":"0'0","reported_seq":197,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.381080+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:55:42.381080+0000","last_peered":"2024-09-04T21:55:42.381080+0000","last_clean":"2024-09-04T21:55:42.381080+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:55:42.381080+0000","last_undegraded":"2024-09-04T21:55:42.381080+0000","last_fullsized":"2024-09-04T21:55:42.381080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"112'583","reported_seq":4467,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.398219+0000","last_change":"2024-09-04T21:55:46.399438+0000","last_active":"2024-09-04T21:55:46.398219+0000","last_peered":"2024-09-04T21:55:46.398219+0000","last_clean":"2024-09-04T21:55:46.398219+0000","last_became_active":"2024-09-04T21:55:41.368545+0000","last_became_peered":"2024-09-04T21:55:41.368545+0000","last_unstale":"2024-09-04T21:55:46.398219+0000","last_undegraded":"2024-09-04T21:55:46.398219+0000","last_fullsized":"2024-09-04T21:55:46.398219+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:46:00.738662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002567097,"stat_sum":{"num_bytes":41564918,"num_objects":28,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":10,"num_read":6180,"num_read_kb":34254,"num_write":839,"num_write_kb":14489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3663317,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.0","version":"0'0","reported_seq":160,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.390583+0000","last_change":"2024-09-04T21:54:48.373076+0000","last_active":"2024-09-04T21:55:44.390583+0000","last_peered":"2024-09-04T21:55:44.390583+0000","last_clean":"2024-09-04T21:55:44.390583+0000","last_became_active":"2024-09-04T21:54:48.372613+0000","last_became_peered":"2024-09-04T21:54:48.372613+0000","last_unstale":"2024-09-04T21:55:44.390583+0000","last_undegraded":"2024-09-04T21:55:44.390583+0000","last_fullsized":"2024-09-04T21:55:44.390583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:48:13.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"110'370","reported_seq":3299,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.395017+0000","last_change":"2024-09-04T21:55:42.395550+0000","last_active":"2024-09-04T21:55:42.395017+0000","last_peered":"2024-09-04T21:55:42.395017+0000","last_clean":"2024-09-04T21:55:42.395017+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:55:42.395017+0000","last_undegraded":"2024-09-04T21:55:42.395017+0000","last_fullsized":"2024-09-04T21:55:42.395017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:02:23.436970+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.021695378000000001,"stat_sum":{"num_bytes":27533052,"num_objects":42,"num_object_clones":7,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":14,"num_read":3770,"num_read_kb":14897,"num_write":344,"num_write_kb":5843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.3","version":"19'1","reported_seq":120,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.381368+0000","last_change":"2024-09-04T21:54:42.676579+0000","last_active":"2024-09-04T21:55:42.381368+0000","last_peered":"2024-09-04T21:55:42.381368+0000","last_clean":"2024-09-04T21:55:42.381368+0000","last_became_active":"2024-09-04T21:54:42.676301+0000","last_became_peered":"2024-09-04T21:54:42.676301+0000","last_unstale":"2024-09-04T21:55:42.381368+0000","last_undegraded":"2024-09-04T21:55:42.381368+0000","last_fullsized":"2024-09-04T21:55:42.381368+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:42:21.618311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":250,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.390482+0000","last_change":"2024-09-04T21:54:48.383973+0000","last_active":"2024-09-04T21:55:44.390482+0000","last_peered":"2024-09-04T21:55:44.390482+0000","last_clean":"2024-09-04T21:55:44.390482+0000","last_became_active":"2024-09-04T21:54:48.374571+0000","last_became_peered":"2024-09-04T21:54:48.374571+0000","last_unstale":"2024-09-04T21:55:44.390482+0000","last_undegraded":"2024-09-04T21:55:44.390482+0000","last_fullsized":"2024-09-04T21:55:44.390482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:54:44.340649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"110'525","reported_seq":3238,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.381005+0000","last_change":"2024-09-04T21:55:42.382922+0000","last_active":"2024-09-04T21:55:42.381005+0000","last_peered":"2024-09-04T21:55:42.381005+0000","last_clean":"2024-09-04T21:55:42.381005+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:55:42.381005+0000","last_undegraded":"2024-09-04T21:55:42.381005+0000","last_fullsized":"2024-09-04T21:55:42.381005+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:39.900338+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0092581030000000002,"stat_sum":{"num_bytes":20541879,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":11,"num_read":4182,"num_read_kb":14977,"num_write":640,"num_write_kb":7058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.9","version":"113'320","reported_seq":1988,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.400462+0000","last_change":"2024-09-04T21:55:46.401353+0000","last_active":"2024-09-04T21:55:46.400462+0000","last_peered":"2024-09-04T21:55:46.400462+0000","last_clean":"2024-09-04T21:55:46.400462+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:55:46.400462+0000","last_undegraded":"2024-09-04T21:55:46.400462+0000","last_fullsized":"2024-09-04T21:55:46.400462+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:45.709975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00447439,"stat_sum":{"num_bytes":13389978,"num_objects":28,"num_object_clones":10,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":13,"num_read":2780,"num_read_kb":16422,"num_write":301,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.a","version":"111'595","reported_seq":2502,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.481082+0000","last_change":"2024-09-04T21:55:43.481082+0000","last_active":"2024-09-04T21:55:43.481082+0000","last_peered":"2024-09-04T21:55:43.481082+0000","last_clean":"2024-09-04T21:55:43.481082+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:55:43.481082+0000","last_undegraded":"2024-09-04T21:55:43.481082+0000","last_fullsized":"2024-09-04T21:55:43.481082+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:36:21.573060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011918899999999999,"stat_sum":{"num_bytes":55379570,"num_objects":33,"num_object_clones":6,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":6,"num_read":3510,"num_read_kb":13263,"num_write":733,"num_write_kb":9480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"b"}]},{"pgid":"3.b","version":"107'333","reported_seq":1871,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.374221+0000","last_change":"2024-09-04T21:55:42.374855+0000","last_active":"2024-09-04T21:55:42.374221+0000","last_peered":"2024-09-04T21:55:42.374221+0000","last_clean":"2024-09-04T21:55:42.374221+0000","last_became_active":"2024-09-04T21:55:38.754456+0000","last_became_peered":"2024-09-04T21:55:38.754456+0000","last_unstale":"2024-09-04T21:55:42.374221+0000","last_undegraded":"2024-09-04T21:55:42.374221+0000","last_fullsized":"2024-09-04T21:55:42.374221+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:33:10.755296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00109541,"stat_sum":{"num_bytes":4155919,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":9,"num_read":2532,"num_read_kb":7561,"num_write":391,"num_write_kb":6199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":1821796,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.c","version":"110'330","reported_seq":3364,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.519032+0000","last_change":"2024-09-04T21:55:42.405904+0000","last_active":"2024-09-04T21:55:42.519032+0000","last_peered":"2024-09-04T21:55:42.519032+0000","last_clean":"2024-09-04T21:55:42.519032+0000","last_became_active":"2024-09-04T21:54:42.679512+0000","last_became_peered":"2024-09-04T21:54:42.679512+0000","last_unstale":"2024-09-04T21:55:42.519032+0000","last_undegraded":"2024-09-04T21:55:42.519032+0000","last_fullsized":"2024-09-04T21:55:42.519032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:13:49.105591+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031746750999999997,"stat_sum":{"num_bytes":34943454,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":3701,"num_read_kb":12906,"num_write":304,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.d","version":"112'397","reported_seq":10994,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.479305+0000","last_change":"2024-09-04T21:55:44.393316+0000","last_active":"2024-09-04T21:55:44.479305+0000","last_peered":"2024-09-04T21:55:44.479305+0000","last_clean":"2024-09-04T21:55:44.479305+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:55:44.479305+0000","last_undegraded":"2024-09-04T21:55:44.479305+0000","last_fullsized":"2024-09-04T21:55:44.479305+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:07:53.736356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024144539999999999,"stat_sum":{"num_bytes":6813786,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":17,"num_read":11855,"num_read_kb":18507,"num_write":387,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.e","version":"112'367","reported_seq":1160,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.468547+0000","last_change":"2024-09-04T21:55:44.393322+0000","last_active":"2024-09-04T21:55:44.468547+0000","last_peered":"2024-09-04T21:55:44.468547+0000","last_clean":"2024-09-04T21:55:44.468547+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:55:44.468547+0000","last_undegraded":"2024-09-04T21:55:44.468547+0000","last_fullsized":"2024-09-04T21:55:44.468547+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:50.196626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022153659999999999,"stat_sum":{"num_bytes":13791115,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":2160,"num_read_kb":12559,"num_write":449,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.f","version":"112'444","reported_seq":11632,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.393871+0000","last_change":"2024-09-04T21:55:44.393871+0000","last_active":"2024-09-04T21:55:44.393871+0000","last_peered":"2024-09-04T21:55:44.393871+0000","last_clean":"2024-09-04T21:55:44.393871+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:55:44.393871+0000","last_undegraded":"2024-09-04T21:55:44.393871+0000","last_fullsized":"2024-09-04T21:55:44.393871+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:08:21.242014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027049600000000002,"stat_sum":{"num_bytes":12247303,"num_objects":32,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":16,"num_read":11453,"num_read_kb":24633,"num_write":441,"num_write_kb":10764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.10","version":"110'743","reported_seq":3101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.566237+0000","last_change":"2024-09-04T21:55:43.481395+0000","last_active":"2024-09-04T21:55:43.566237+0000","last_peered":"2024-09-04T21:55:43.566237+0000","last_clean":"2024-09-04T21:55:43.566237+0000","last_became_active":"2024-09-04T21:54:51.358608+0000","last_became_peered":"2024-09-04T21:54:51.358608+0000","last_unstale":"2024-09-04T21:55:43.566237+0000","last_undegraded":"2024-09-04T21:55:43.566237+0000","last_fullsized":"2024-09-04T21:55:43.566237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:02:48.524208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00149124,"stat_sum":{"num_bytes":8946467,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":14,"num_read":5071,"num_read_kb":12574,"num_write":959,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3966726,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"b"}]},{"pgid":"3.11","version":"110'514","reported_seq":5713,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.392378+0000","last_change":"2024-09-04T21:55:44.392692+0000","last_active":"2024-09-04T21:55:44.392378+0000","last_peered":"2024-09-04T21:55:44.392378+0000","last_clean":"2024-09-04T21:55:44.392378+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:55:44.392378+0000","last_undegraded":"2024-09-04T21:55:44.392378+0000","last_fullsized":"2024-09-04T21:55:44.392378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:51:29.979040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017938100000000001,"stat_sum":{"num_bytes":34095845,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":11,"num_read":6624,"num_read_kb":44010,"num_write":638,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.12","version":"112'675","reported_seq":13275,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.394549+0000","last_change":"2024-09-04T21:55:44.394549+0000","last_active":"2024-09-04T21:55:44.394549+0000","last_peered":"2024-09-04T21:55:44.394549+0000","last_clean":"2024-09-04T21:55:44.394549+0000","last_became_active":"2024-09-04T21:54:51.349872+0000","last_became_peered":"2024-09-04T21:54:51.349872+0000","last_unstale":"2024-09-04T21:55:44.394549+0000","last_undegraded":"2024-09-04T21:55:44.394549+0000","last_fullsized":"2024-09-04T21:55:44.394549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:51:11.098352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032305300000000001,"stat_sum":{"num_bytes":36868476,"num_objects":43,"num_object_clones":15,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":13,"num_read":14380,"num_read_kb":37893,"num_write":857,"num_write_kb":9458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":7361343,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.13","version":"112'937","reported_seq":2393,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391380+0000","last_change":"2024-09-04T21:55:44.391380+0000","last_active":"2024-09-04T21:55:44.391380+0000","last_peered":"2024-09-04T21:55:44.391380+0000","last_clean":"2024-09-04T21:55:44.391380+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:55:44.391380+0000","last_undegraded":"2024-09-04T21:55:44.391380+0000","last_fullsized":"2024-09-04T21:55:44.391380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:11:09.610637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066224800000000005,"stat_sum":{"num_bytes":19556701,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":9,"num_read":4657,"num_read_kb":19276,"num_write":1445,"num_write_kb":7289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":630157856,"num_objects":968,"num_object_clones":224,"num_object_copies":1936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":369,"num_read":186238,"num_read_kb":668705,"num_write":19586,"num_write_kb":314125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1230,"num_bytes_recovered":57197824,"num_keys_recovered":237,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33067008,"data_stored":293097140,"data_compressed":15605579,"data_compressed_allocated":26832896,"data_compressed_original":289186143,"omap_allocated":14003,"internal_metadata":0},"log_size":15820,"ondisk_log_size":15820,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":18583,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509469,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47116,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324724,"statfs":{"total":96636764160,"available":96588517376,"internally_reserved":0,"allocated":3530752,"data_stored":19814688,"data_compressed":971470,"data_compressed_allocated":1736704,"data_compressed_original":18922702,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566083,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40964,"kb_used_data":7724,"kb_used_omap":33,"kb_used_meta":33182,"kb_avail":94330876,"statfs":{"total":96636764160,"available":96594817024,"internally_reserved":0,"allocated":7909376,"data_stored":58512241,"data_compressed":3018143,"data_compressed_allocated":5267456,"data_compressed_original":57079943,"omap_allocated":34021,"internal_metadata":33979163},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509469,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542172,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62004,"kb_used_data":8600,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309836,"statfs":{"total":96636764160,"available":96573272064,"internally_reserved":0,"allocated":8806400,"data_stored":67355297,"data_compressed":3566953,"data_compressed_allocated":6082560,"data_compressed_original":65804640,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43160,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328680,"statfs":{"total":96636764160,"available":96592568320,"internally_reserved":0,"allocated":3739648,"data_stored":22831951,"data_compressed":1159897,"data_compressed_allocated":1982464,"data_compressed_original":21942227,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430224,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35732,"kb_used_data":7740,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94336108,"statfs":{"total":96636764160,"available":96600174592,"internally_reserved":0,"allocated":7925760,"data_stored":58644907,"data_compressed":2871268,"data_compressed_allocated":5410816,"data_compressed_original":57345265,"omap_allocated":4394,"internal_metadata":28634838},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58916,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312924,"statfs":{"total":96636764160,"available":96576434176,"internally_reserved":0,"allocated":5619712,"data_stored":37694679,"data_compressed":2028728,"data_compressed_allocated":3444736,"data_compressed_original":36503739,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":35643695,"data_compressed":2027336,"data_compressed_allocated":3260416,"data_compressed_original":35162080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2371584,"data_stored":21921646,"data_compressed":1149376,"data_compressed_allocated":1945600,"data_compressed_original":21655507,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7225344,"data_stored":64748907,"data_compressed":3518857,"data_compressed_allocated":5902336,"data_compressed_original":63871328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6557696,"data_stored":57872892,"data_compressed":3018143,"data_compressed_allocated":5267456,"data_compressed_original":57079943,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4075520,"data_stored":36462301,"data_compressed":1910825,"data_compressed_allocated":3289088,"data_compressed_original":35922107,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2170880,"data_stored":18603657,"data_compressed":964735,"data_compressed_allocated":1699840,"data_compressed_original":18332878,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6303744,"data_stored":56156186,"data_compressed":2824580,"data_compressed_allocated":5181440,"data_compressed_original":55526641,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:55:47.770 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:55:47.770 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.770 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:47.827 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:47.823+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:47.916 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:47.911+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:47.946 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:47.946 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:47.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:48.017 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:48.011+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:48.118 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:48.111+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:48.125 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:48.125 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:48.133 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:55:48.134 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:55:48.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:48.153 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-04T21:55:47.412320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630748243,"num_objects":972,"num_object_clones":224,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":369,"num_read":186304,"num_read_kb":668760,"num_write":19689,"num_write_kb":315755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1232,"num_bytes_recovered":57788192,"num_keys_recovered":237,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15895,"ondisk_log_size":15895,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":287892,"kb_used_data":36652,"kb_used_omap":43,"kb_used_meta":251028,"kb_avail":565943148,"statfs":{"total":579820584960,"available":579525783552,"internally_reserved":0,"allocated":37531648,"data_stored":264853763,"data_compressed":13616459,"data_compressed_allocated":23924736,"data_compressed_original":257598516,"omap_allocated":44792,"internal_metadata":257052936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-155716629,"num_objects":-157,"num_object_clones":-60,"num_object_copies":-314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-157,"num_whiteouts":-46,"num_read":696,"num_read_kb":604,"num_write":158,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":124562,"num_keys_recovered":18,"num_objects_omap":-3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030179"},"pg_stats":[{"pgid":"3.1f","version":"110'730","reported_seq":2662,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.399885+0000","last_change":"2024-09-04T21:55:42.400305+0000","last_active":"2024-09-04T21:55:42.399885+0000","last_peered":"2024-09-04T21:55:42.399885+0000","last_clean":"2024-09-04T21:55:42.399885+0000","last_became_active":"2024-09-04T21:54:51.350505+0000","last_became_peered":"2024-09-04T21:54:51.350505+0000","last_unstale":"2024-09-04T21:55:42.399885+0000","last_undegraded":"2024-09-04T21:55:42.399885+0000","last_fullsized":"2024-09-04T21:55:42.399885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:42:08.481832+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.018428050000000001,"stat_sum":{"num_bytes":15466997,"num_objects":21,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":12,"num_read":4923,"num_read_kb":7638,"num_write":1065,"num_write_kb":4569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":330234,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.1e","version":"110'355","reported_seq":3289,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.465147+0000","last_change":"2024-09-04T21:55:42.399425+0000","last_active":"2024-09-04T21:55:42.465147+0000","last_peered":"2024-09-04T21:55:42.465147+0000","last_clean":"2024-09-04T21:55:42.465147+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:55:42.465147+0000","last_undegraded":"2024-09-04T21:55:42.465147+0000","last_fullsized":"2024-09-04T21:55:42.465147+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:04:02.685368+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.025135099000000001,"stat_sum":{"num_bytes":15930092,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":13,"num_read":3609,"num_read_kb":13267,"num_write":334,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.1d","version":"112'508","reported_seq":18817,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.401624+0000","last_change":"2024-09-04T21:55:46.402623+0000","last_active":"2024-09-04T21:55:46.401624+0000","last_peered":"2024-09-04T21:55:46.401624+0000","last_clean":"2024-09-04T21:55:46.401624+0000","last_became_active":"2024-09-04T21:55:38.745373+0000","last_became_peered":"2024-09-04T21:55:38.745373+0000","last_unstale":"2024-09-04T21:55:46.401624+0000","last_undegraded":"2024-09-04T21:55:46.401624+0000","last_fullsized":"2024-09-04T21:55:46.401624+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:05:21.791227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050457210000000004,"stat_sum":{"num_bytes":33053972,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":8,"num_read":19674,"num_read_kb":23021,"num_write":691,"num_write_kb":9166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":911480,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.1c","version":"114'354","reported_seq":2169,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.550746+0000","last_change":"2024-09-04T21:55:46.404207+0000","last_active":"2024-09-04T21:55:46.550746+0000","last_peered":"2024-09-04T21:55:46.550746+0000","last_clean":"2024-09-04T21:55:46.550746+0000","last_became_active":"2024-09-04T21:55:38.745462+0000","last_became_peered":"2024-09-04T21:55:38.745462+0000","last_unstale":"2024-09-04T21:55:46.550746+0000","last_undegraded":"2024-09-04T21:55:46.550746+0000","last_fullsized":"2024-09-04T21:55:46.550746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:37:38.424780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006521186,"stat_sum":{"num_bytes":18878302,"num_objects":26,"num_object_clones":7,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":12,"num_read":2087,"num_read_kb":21010,"num_write":343,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.1b","version":"113'272","reported_seq":1491,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.400294+0000","last_change":"2024-09-04T21:55:46.401369+0000","last_active":"2024-09-04T21:55:46.400294+0000","last_peered":"2024-09-04T21:55:46.400294+0000","last_clean":"2024-09-04T21:55:46.400294+0000","last_became_active":"2024-09-04T21:55:38.763671+0000","last_became_peered":"2024-09-04T21:55:38.763671+0000","last_unstale":"2024-09-04T21:55:46.400294+0000","last_undegraded":"2024-09-04T21:55:46.400294+0000","last_fullsized":"2024-09-04T21:55:46.400294+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T21:58:18.921704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046880020000000001,"stat_sum":{"num_bytes":5396272,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":11,"num_read":2251,"num_read_kb":17532,"num_write":263,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2186006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.1a","version":"114'907","reported_seq":16937,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.630386+0000","last_change":"2024-09-04T21:55:46.398869+0000","last_active":"2024-09-04T21:55:46.630386+0000","last_peered":"2024-09-04T21:55:46.630386+0000","last_clean":"2024-09-04T21:55:46.630386+0000","last_became_active":"2024-09-04T21:55:41.365854+0000","last_became_peered":"2024-09-04T21:55:41.365854+0000","last_unstale":"2024-09-04T21:55:46.630386+0000","last_undegraded":"2024-09-04T21:55:46.630386+0000","last_fullsized":"2024-09-04T21:55:46.630386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":907,"log_dups_size":0,"ondisk_log_size":907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:59:36.861119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002204396,"stat_sum":{"num_bytes":28526938,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":14,"num_read":19752,"num_read_kb":56989,"num_write":1186,"num_write_kb":17244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1484312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.19","version":"110'1161","reported_seq":2921,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.533455+0000","last_change":"2024-09-04T21:55:42.415259+0000","last_active":"2024-09-04T21:55:42.533455+0000","last_peered":"2024-09-04T21:55:42.533455+0000","last_clean":"2024-09-04T21:55:42.533455+0000","last_became_active":"2024-09-04T21:55:38.757506+0000","last_became_peered":"2024-09-04T21:55:38.757506+0000","last_unstale":"2024-09-04T21:55:42.533455+0000","last_undegraded":"2024-09-04T21:55:42.533455+0000","last_fullsized":"2024-09-04T21:55:42.533455+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1161,"log_dups_size":0,"ondisk_log_size":1161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:25:33.800130+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.040984010000000001,"stat_sum":{"num_bytes":14223921,"num_objects":26,"num_object_clones":5,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":6610,"num_read_kb":17001,"num_write":1856,"num_write_kb":6539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":37560,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.18","version":"110'339","reported_seq":3915,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.385503+0000","last_change":"2024-09-04T21:55:42.387167+0000","last_active":"2024-09-04T21:55:42.385503+0000","last_peered":"2024-09-04T21:55:42.385503+0000","last_clean":"2024-09-04T21:55:42.385503+0000","last_became_active":"2024-09-04T21:54:51.349573+0000","last_became_peered":"2024-09-04T21:54:51.349573+0000","last_unstale":"2024-09-04T21:55:42.385503+0000","last_undegraded":"2024-09-04T21:55:42.385503+0000","last_fullsized":"2024-09-04T21:55:42.385503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:59:13.025001+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.013391735,"stat_sum":{"num_bytes":22865144,"num_objects":24,"num_object_clones":6,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":9,"num_read":4669,"num_read_kb":25777,"num_write":348,"num_write_kb":8113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3632211,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.17","version":"110'840","reported_seq":2700,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.579120+0000","last_change":"2024-09-04T21:55:42.416831+0000","last_active":"2024-09-04T21:55:42.579120+0000","last_peered":"2024-09-04T21:55:42.579120+0000","last_clean":"2024-09-04T21:55:42.579120+0000","last_became_active":"2024-09-04T21:54:51.352376+0000","last_became_peered":"2024-09-04T21:54:51.352376+0000","last_unstale":"2024-09-04T21:55:42.579120+0000","last_undegraded":"2024-09-04T21:55:42.579120+0000","last_fullsized":"2024-09-04T21:55:42.579120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:20:10.043365+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.042177032000000003,"stat_sum":{"num_bytes":31371140,"num_objects":35,"num_object_clones":2,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":14,"num_read":5136,"num_read_kb":21920,"num_write":1211,"num_write_kb":7086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3732324,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.16","version":"112'340","reported_seq":2511,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391858+0000","last_change":"2024-09-04T21:55:44.391858+0000","last_active":"2024-09-04T21:55:44.391858+0000","last_peered":"2024-09-04T21:55:44.391858+0000","last_clean":"2024-09-04T21:55:44.391858+0000","last_became_active":"2024-09-04T21:54:42.679389+0000","last_became_peered":"2024-09-04T21:54:42.679389+0000","last_unstale":"2024-09-04T21:55:44.391858+0000","last_undegraded":"2024-09-04T21:55:44.391858+0000","last_fullsized":"2024-09-04T21:55:44.391858+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:37:42.001276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010147750000000001,"stat_sum":{"num_bytes":13184683,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":8,"num_read":2897,"num_read_kb":16809,"num_write":322,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.15","version":"110'362","reported_seq":2937,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391127+0000","last_change":"2024-09-04T21:55:44.391562+0000","last_active":"2024-09-04T21:55:44.391127+0000","last_peered":"2024-09-04T21:55:44.391127+0000","last_clean":"2024-09-04T21:55:44.391127+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:55:44.391127+0000","last_undegraded":"2024-09-04T21:55:44.391127+0000","last_fullsized":"2024-09-04T21:55:44.391127+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:27:03.130370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076071899999999998,"stat_sum":{"num_bytes":18646762,"num_objects":43,"num_object_clones":10,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":22,"num_read":3202,"num_read_kb":27791,"num_write":383,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.14","version":"114'316","reported_seq":1850,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.400172+0000","last_change":"2024-09-04T21:55:46.400172+0000","last_active":"2024-09-04T21:55:46.400172+0000","last_peered":"2024-09-04T21:55:46.400172+0000","last_clean":"2024-09-04T21:55:46.400172+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:55:46.400172+0000","last_undegraded":"2024-09-04T21:55:46.400172+0000","last_fullsized":"2024-09-04T21:55:46.400172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:28:21.405856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033055609999999998,"stat_sum":{"num_bytes":4838177,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":7,"num_read":1562,"num_read_kb":6375,"num_write":344,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.7","version":"0'0","reported_seq":208,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.479511+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:55:43.479511+0000","last_peered":"2024-09-04T21:55:43.479511+0000","last_clean":"2024-09-04T21:55:43.479511+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:55:43.479511+0000","last_undegraded":"2024-09-04T21:55:43.479511+0000","last_fullsized":"2024-09-04T21:55:43.479511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"114'412","reported_seq":2234,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.605364+0000","last_change":"2024-09-04T21:55:46.402588+0000","last_active":"2024-09-04T21:55:46.605364+0000","last_peered":"2024-09-04T21:55:46.605364+0000","last_clean":"2024-09-04T21:55:46.605364+0000","last_became_active":"2024-09-04T21:55:41.369871+0000","last_became_peered":"2024-09-04T21:55:41.369871+0000","last_unstale":"2024-09-04T21:55:46.605364+0000","last_undegraded":"2024-09-04T21:55:46.605364+0000","last_fullsized":"2024-09-04T21:55:46.605364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:28:27.105299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051780849999999998,"stat_sum":{"num_bytes":4136554,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":3134,"num_read_kb":5309,"num_write":370,"num_write_kb":7369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":946773,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.6","version":"0'0","reported_seq":170,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.479625+0000","last_change":"2024-09-04T21:55:38.759143+0000","last_active":"2024-09-04T21:55:43.479625+0000","last_peered":"2024-09-04T21:55:43.479625+0000","last_clean":"2024-09-04T21:55:43.479625+0000","last_became_active":"2024-09-04T21:55:38.758298+0000","last_became_peered":"2024-09-04T21:55:38.758298+0000","last_unstale":"2024-09-04T21:55:43.479625+0000","last_undegraded":"2024-09-04T21:55:43.479625+0000","last_fullsized":"2024-09-04T21:55:43.479625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:32:19.422630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"110'296","reported_seq":3196,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.419249+0000","last_change":"2024-09-04T21:55:42.419566+0000","last_active":"2024-09-04T21:55:42.419249+0000","last_peered":"2024-09-04T21:55:42.419249+0000","last_clean":"2024-09-04T21:55:42.419249+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:55:42.419249+0000","last_undegraded":"2024-09-04T21:55:42.419249+0000","last_fullsized":"2024-09-04T21:55:42.419249+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:32:58.519392+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.044636157000000003,"stat_sum":{"num_bytes":10387018,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":10,"num_read":3238,"num_read_kb":21928,"num_write":326,"num_write_kb":14374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391031+0000","last_change":"2024-09-04T21:55:38.759967+0000","last_active":"2024-09-04T21:55:44.391031+0000","last_peered":"2024-09-04T21:55:44.391031+0000","last_clean":"2024-09-04T21:55:44.391031+0000","last_became_active":"2024-09-04T21:55:38.759102+0000","last_became_peered":"2024-09-04T21:55:38.759102+0000","last_unstale":"2024-09-04T21:55:44.391031+0000","last_undegraded":"2024-09-04T21:55:44.391031+0000","last_fullsized":"2024-09-04T21:55:44.391031+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:21:11.118581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"110'292","reported_seq":1619,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.498765+0000","last_change":"2024-09-04T21:55:42.389843+0000","last_active":"2024-09-04T21:55:42.498765+0000","last_peered":"2024-09-04T21:55:42.498765+0000","last_clean":"2024-09-04T21:55:42.498765+0000","last_became_active":"2024-09-04T21:54:13.482562+0000","last_became_peered":"2024-09-04T21:54:13.482562+0000","last_unstale":"2024-09-04T21:55:42.498765+0000","last_undegraded":"2024-09-04T21:55:42.498765+0000","last_fullsized":"2024-09-04T21:55:42.498765+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:31:27.322233+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0087944479999999998,"stat_sum":{"num_bytes":6730531,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":9,"num_read":1744,"num_read_kb":7801,"num_write":260,"num_write_kb":3585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":344612,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.4","version":"0'0","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.390625+0000","last_change":"2024-09-04T21:54:48.372640+0000","last_active":"2024-09-04T21:55:44.390625+0000","last_peered":"2024-09-04T21:55:44.390625+0000","last_clean":"2024-09-04T21:55:44.390625+0000","last_became_active":"2024-09-04T21:54:48.372106+0000","last_became_peered":"2024-09-04T21:54:48.372106+0000","last_unstale":"2024-09-04T21:55:44.390625+0000","last_undegraded":"2024-09-04T21:55:44.390625+0000","last_fullsized":"2024-09-04T21:55:44.390625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:18:59.210273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"110'478","reported_seq":11459,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.409941+0000","last_change":"2024-09-04T21:55:42.410423+0000","last_active":"2024-09-04T21:55:42.409941+0000","last_peered":"2024-09-04T21:55:42.409941+0000","last_clean":"2024-09-04T21:55:42.409941+0000","last_became_active":"2024-09-04T21:54:42.679683+0000","last_became_peered":"2024-09-04T21:54:42.679683+0000","last_unstale":"2024-09-04T21:55:42.409941+0000","last_undegraded":"2024-09-04T21:55:42.409941+0000","last_fullsized":"2024-09-04T21:55:42.409941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:50:17.531826+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.035823607,"stat_sum":{"num_bytes":28978471,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":10,"num_read":12262,"num_read_kb":24996,"num_write":542,"num_write_kb":14132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.3","version":"114'404","reported_seq":3555,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.577396+0000","last_change":"2024-09-04T21:55:46.404031+0000","last_active":"2024-09-04T21:55:46.577396+0000","last_peered":"2024-09-04T21:55:46.577396+0000","last_clean":"2024-09-04T21:55:46.577396+0000","last_became_active":"2024-09-04T21:55:41.371293+0000","last_became_peered":"2024-09-04T21:55:41.371293+0000","last_unstale":"2024-09-04T21:55:46.577396+0000","last_undegraded":"2024-09-04T21:55:46.577396+0000","last_fullsized":"2024-09-04T21:55:46.577396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:16:32.630273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061329899999999996,"stat_sum":{"num_bytes":13250427,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":3237,"num_read_kb":16214,"num_write":460,"num_write_kb":12348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":2080930,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.2","version":"21'2","reported_seq":161,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.479355+0000","last_change":"2024-09-04T21:54:42.675259+0000","last_active":"2024-09-04T21:55:43.479355+0000","last_peered":"2024-09-04T21:55:43.479355+0000","last_clean":"2024-09-04T21:55:43.479355+0000","last_became_active":"2024-09-04T21:54:42.674803+0000","last_became_peered":"2024-09-04T21:54:42.674803+0000","last_unstale":"2024-09-04T21:55:43.479355+0000","last_undegraded":"2024-09-04T21:55:43.479355+0000","last_fullsized":"2024-09-04T21:55:43.479355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:09:21.260817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"110'321","reported_seq":2902,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.394233+0000","last_change":"2024-09-04T21:55:42.394233+0000","last_active":"2024-09-04T21:55:42.394233+0000","last_peered":"2024-09-04T21:55:42.394233+0000","last_clean":"2024-09-04T21:55:42.394233+0000","last_became_active":"2024-09-04T21:54:16.342229+0000","last_became_peered":"2024-09-04T21:54:16.342229+0000","last_unstale":"2024-09-04T21:55:42.394233+0000","last_undegraded":"2024-09-04T21:55:42.394233+0000","last_fullsized":"2024-09-04T21:55:42.394233+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:44:44.010352+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.013247919,"stat_sum":{"num_bytes":14463992,"num_objects":40,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":15,"num_read":3396,"num_read_kb":53595,"num_write":554,"num_write_kb":21343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":539785,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.1","version":"0'0","reported_seq":197,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.381080+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:55:42.381080+0000","last_peered":"2024-09-04T21:55:42.381080+0000","last_clean":"2024-09-04T21:55:42.381080+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:55:42.381080+0000","last_undegraded":"2024-09-04T21:55:42.381080+0000","last_fullsized":"2024-09-04T21:55:42.381080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"112'583","reported_seq":4467,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.398219+0000","last_change":"2024-09-04T21:55:46.399438+0000","last_active":"2024-09-04T21:55:46.398219+0000","last_peered":"2024-09-04T21:55:46.398219+0000","last_clean":"2024-09-04T21:55:46.398219+0000","last_became_active":"2024-09-04T21:55:41.368545+0000","last_became_peered":"2024-09-04T21:55:41.368545+0000","last_unstale":"2024-09-04T21:55:46.398219+0000","last_undegraded":"2024-09-04T21:55:46.398219+0000","last_fullsized":"2024-09-04T21:55:46.398219+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:46:00.738662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002567097,"stat_sum":{"num_bytes":41564918,"num_objects":28,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":10,"num_read":6180,"num_read_kb":34254,"num_write":839,"num_write_kb":14489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3663317,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"2.0","version":"0'0","reported_seq":160,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.390583+0000","last_change":"2024-09-04T21:54:48.373076+0000","last_active":"2024-09-04T21:55:44.390583+0000","last_peered":"2024-09-04T21:55:44.390583+0000","last_clean":"2024-09-04T21:55:44.390583+0000","last_became_active":"2024-09-04T21:54:48.372613+0000","last_became_peered":"2024-09-04T21:54:48.372613+0000","last_unstale":"2024-09-04T21:55:44.390583+0000","last_undegraded":"2024-09-04T21:55:44.390583+0000","last_fullsized":"2024-09-04T21:55:44.390583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:48:13.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"110'370","reported_seq":3299,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.395017+0000","last_change":"2024-09-04T21:55:42.395550+0000","last_active":"2024-09-04T21:55:42.395017+0000","last_peered":"2024-09-04T21:55:42.395017+0000","last_clean":"2024-09-04T21:55:42.395017+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:55:42.395017+0000","last_undegraded":"2024-09-04T21:55:42.395017+0000","last_fullsized":"2024-09-04T21:55:42.395017+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:02:23.436970+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.021695378000000001,"stat_sum":{"num_bytes":27533052,"num_objects":42,"num_object_clones":7,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":14,"num_read":3770,"num_read_kb":14897,"num_write":344,"num_write_kb":5843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"2.3","version":"19'1","reported_seq":120,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.381368+0000","last_change":"2024-09-04T21:54:42.676579+0000","last_active":"2024-09-04T21:55:42.381368+0000","last_peered":"2024-09-04T21:55:42.381368+0000","last_clean":"2024-09-04T21:55:42.381368+0000","last_became_active":"2024-09-04T21:54:42.676301+0000","last_became_peered":"2024-09-04T21:54:42.676301+0000","last_unstale":"2024-09-04T21:55:42.381368+0000","last_undegraded":"2024-09-04T21:55:42.381368+0000","last_fullsized":"2024-09-04T21:55:42.381368+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:42:21.618311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":250,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.390482+0000","last_change":"2024-09-04T21:54:48.383973+0000","last_active":"2024-09-04T21:55:44.390482+0000","last_peered":"2024-09-04T21:55:44.390482+0000","last_clean":"2024-09-04T21:55:44.390482+0000","last_became_active":"2024-09-04T21:54:48.374571+0000","last_became_peered":"2024-09-04T21:54:48.374571+0000","last_unstale":"2024-09-04T21:55:44.390482+0000","last_undegraded":"2024-09-04T21:55:44.390482+0000","last_fullsized":"2024-09-04T21:55:44.390482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:54:44.340649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"110'525","reported_seq":3238,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.381005+0000","last_change":"2024-09-04T21:55:42.382922+0000","last_active":"2024-09-04T21:55:42.381005+0000","last_peered":"2024-09-04T21:55:42.381005+0000","last_clean":"2024-09-04T21:55:42.381005+0000","last_became_active":"2024-09-04T21:54:34.606317+0000","last_became_peered":"2024-09-04T21:54:34.606317+0000","last_unstale":"2024-09-04T21:55:42.381005+0000","last_undegraded":"2024-09-04T21:55:42.381005+0000","last_fullsized":"2024-09-04T21:55:42.381005+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:39.900338+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0092581030000000002,"stat_sum":{"num_bytes":20541879,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":11,"num_read":4182,"num_read_kb":14977,"num_write":640,"num_write_kb":7058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1517666,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.9","version":"113'320","reported_seq":1988,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-04T21:55:46.400462+0000","last_change":"2024-09-04T21:55:46.401353+0000","last_active":"2024-09-04T21:55:46.400462+0000","last_peered":"2024-09-04T21:55:46.400462+0000","last_clean":"2024-09-04T21:55:46.400462+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:55:46.400462+0000","last_undegraded":"2024-09-04T21:55:46.400462+0000","last_fullsized":"2024-09-04T21:55:46.400462+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:45.709975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00447439,"stat_sum":{"num_bytes":13389978,"num_objects":28,"num_object_clones":10,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":13,"num_read":2780,"num_read_kb":16422,"num_write":301,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"e"}]},{"pgid":"3.a","version":"111'595","reported_seq":2502,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.481082+0000","last_change":"2024-09-04T21:55:43.481082+0000","last_active":"2024-09-04T21:55:43.481082+0000","last_peered":"2024-09-04T21:55:43.481082+0000","last_clean":"2024-09-04T21:55:43.481082+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:55:43.481082+0000","last_undegraded":"2024-09-04T21:55:43.481082+0000","last_fullsized":"2024-09-04T21:55:43.481082+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:36:21.573060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011918899999999999,"stat_sum":{"num_bytes":55379570,"num_objects":33,"num_object_clones":6,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":6,"num_read":3510,"num_read_kb":13263,"num_write":733,"num_write_kb":9480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"b"}]},{"pgid":"3.b","version":"107'333","reported_seq":1871,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.374221+0000","last_change":"2024-09-04T21:55:42.374855+0000","last_active":"2024-09-04T21:55:42.374221+0000","last_peered":"2024-09-04T21:55:42.374221+0000","last_clean":"2024-09-04T21:55:42.374221+0000","last_became_active":"2024-09-04T21:55:38.754456+0000","last_became_peered":"2024-09-04T21:55:38.754456+0000","last_unstale":"2024-09-04T21:55:42.374221+0000","last_undegraded":"2024-09-04T21:55:42.374221+0000","last_fullsized":"2024-09-04T21:55:42.374221+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:33:10.755296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00109541,"stat_sum":{"num_bytes":4155919,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":9,"num_read":2532,"num_read_kb":7561,"num_write":391,"num_write_kb":6199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":1821796,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.c","version":"110'330","reported_seq":3364,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-04T21:55:42.519032+0000","last_change":"2024-09-04T21:55:42.405904+0000","last_active":"2024-09-04T21:55:42.519032+0000","last_peered":"2024-09-04T21:55:42.519032+0000","last_clean":"2024-09-04T21:55:42.519032+0000","last_became_active":"2024-09-04T21:54:42.679512+0000","last_became_peered":"2024-09-04T21:54:42.679512+0000","last_unstale":"2024-09-04T21:55:42.519032+0000","last_undegraded":"2024-09-04T21:55:42.519032+0000","last_fullsized":"2024-09-04T21:55:42.519032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:13:49.105591+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.031746750999999997,"stat_sum":{"num_bytes":34943454,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":3701,"num_read_kb":12906,"num_write":304,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"a"}]},{"pgid":"3.d","version":"112'397","reported_seq":10994,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.479305+0000","last_change":"2024-09-04T21:55:44.393316+0000","last_active":"2024-09-04T21:55:44.479305+0000","last_peered":"2024-09-04T21:55:44.479305+0000","last_clean":"2024-09-04T21:55:44.479305+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:55:44.479305+0000","last_undegraded":"2024-09-04T21:55:44.479305+0000","last_fullsized":"2024-09-04T21:55:44.479305+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:07:53.736356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024144539999999999,"stat_sum":{"num_bytes":6813786,"num_objects":31,"num_object_clones":8,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":17,"num_read":11855,"num_read_kb":18507,"num_write":387,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.e","version":"112'367","reported_seq":1160,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.468547+0000","last_change":"2024-09-04T21:55:44.393322+0000","last_active":"2024-09-04T21:55:44.468547+0000","last_peered":"2024-09-04T21:55:44.468547+0000","last_clean":"2024-09-04T21:55:44.468547+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:55:44.468547+0000","last_undegraded":"2024-09-04T21:55:44.468547+0000","last_fullsized":"2024-09-04T21:55:44.468547+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:50.196626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022153659999999999,"stat_sum":{"num_bytes":13791115,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":11,"num_read":2160,"num_read_kb":12559,"num_write":449,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.f","version":"112'444","reported_seq":11632,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.393871+0000","last_change":"2024-09-04T21:55:44.393871+0000","last_active":"2024-09-04T21:55:44.393871+0000","last_peered":"2024-09-04T21:55:44.393871+0000","last_clean":"2024-09-04T21:55:44.393871+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:55:44.393871+0000","last_undegraded":"2024-09-04T21:55:44.393871+0000","last_fullsized":"2024-09-04T21:55:44.393871+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:08:21.242014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027049600000000002,"stat_sum":{"num_bytes":12247303,"num_objects":32,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":16,"num_read":11453,"num_read_kb":24633,"num_write":441,"num_write_kb":10764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.10","version":"110'743","reported_seq":3101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-04T21:55:43.566237+0000","last_change":"2024-09-04T21:55:43.481395+0000","last_active":"2024-09-04T21:55:43.566237+0000","last_peered":"2024-09-04T21:55:43.566237+0000","last_clean":"2024-09-04T21:55:43.566237+0000","last_became_active":"2024-09-04T21:54:51.358608+0000","last_became_peered":"2024-09-04T21:54:51.358608+0000","last_unstale":"2024-09-04T21:55:43.566237+0000","last_undegraded":"2024-09-04T21:55:43.566237+0000","last_fullsized":"2024-09-04T21:55:43.566237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:02:48.524208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00149124,"stat_sum":{"num_bytes":8946467,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":14,"num_read":5071,"num_read_kb":12574,"num_write":959,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3966726,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"b"}]},{"pgid":"3.11","version":"110'514","reported_seq":5713,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.392378+0000","last_change":"2024-09-04T21:55:44.392692+0000","last_active":"2024-09-04T21:55:44.392378+0000","last_peered":"2024-09-04T21:55:44.392378+0000","last_clean":"2024-09-04T21:55:44.392378+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:55:44.392378+0000","last_undegraded":"2024-09-04T21:55:44.392378+0000","last_fullsized":"2024-09-04T21:55:44.392378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:51:29.979040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017938100000000001,"stat_sum":{"num_bytes":34095845,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":11,"num_read":6624,"num_read_kb":44010,"num_write":638,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.12","version":"112'675","reported_seq":13275,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.394549+0000","last_change":"2024-09-04T21:55:44.394549+0000","last_active":"2024-09-04T21:55:44.394549+0000","last_peered":"2024-09-04T21:55:44.394549+0000","last_clean":"2024-09-04T21:55:44.394549+0000","last_became_active":"2024-09-04T21:54:51.349872+0000","last_became_peered":"2024-09-04T21:54:51.349872+0000","last_unstale":"2024-09-04T21:55:44.394549+0000","last_undegraded":"2024-09-04T21:55:44.394549+0000","last_fullsized":"2024-09-04T21:55:44.394549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:51:11.098352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032305300000000001,"stat_sum":{"num_bytes":36868476,"num_objects":43,"num_object_clones":15,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":13,"num_read":14380,"num_read_kb":37893,"num_write":857,"num_write_kb":9458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":7361343,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]},{"pgid":"3.13","version":"112'937","reported_seq":2393,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-04T21:55:44.391380+0000","last_change":"2024-09-04T21:55:44.391380+0000","last_active":"2024-09-04T21:55:44.391380+0000","last_peered":"2024-09-04T21:55:44.391380+0000","last_clean":"2024-09-04T21:55:44.391380+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:55:44.391380+0000","last_undegraded":"2024-09-04T21:55:44.391380+0000","last_fullsized":"2024-09-04T21:55:44.391380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:11:09.610637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066224800000000005,"stat_sum":{"num_bytes":19556701,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":9,"num_read":4657,"num_read_kb":19276,"num_write":1445,"num_write_kb":7289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"c"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":630157856,"num_objects":968,"num_object_clones":224,"num_object_copies":1936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":369,"num_read":186238,"num_read_kb":668705,"num_write":19586,"num_write_kb":314125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1230,"num_bytes_recovered":57197824,"num_keys_recovered":237,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33067008,"data_stored":293097140,"data_compressed":15605579,"data_compressed_allocated":26832896,"data_compressed_original":289186143,"omap_allocated":14003,"internal_metadata":0},"log_size":15820,"ondisk_log_size":15820,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":18583,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509469,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47116,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324724,"statfs":{"total":96636764160,"available":96588517376,"internally_reserved":0,"allocated":3530752,"data_stored":19814688,"data_compressed":971470,"data_compressed_allocated":1736704,"data_compressed_original":18922702,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566083,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40964,"kb_used_data":7724,"kb_used_omap":33,"kb_used_meta":33182,"kb_avail":94330876,"statfs":{"total":96636764160,"available":96594817024,"internally_reserved":0,"allocated":7909376,"data_stored":58512241,"data_compressed":3018143,"data_compressed_allocated":5267456,"data_compressed_original":57079943,"omap_allocated":34021,"internal_metadata":33979163},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":64424509469,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542172,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62004,"kb_used_data":8600,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309836,"statfs":{"total":96636764160,"available":96573272064,"internally_reserved":0,"allocated":8806400,"data_stored":67355297,"data_compressed":3566953,"data_compressed_allocated":6082560,"data_compressed_original":65804640,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43160,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328680,"statfs":{"total":96636764160,"available":96592568320,"internally_reserved":0,"allocated":3739648,"data_stored":22831951,"data_compressed":1159897,"data_compressed_allocated":1982464,"data_compressed_original":21942227,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430224,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35732,"kb_used_data":7740,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94336108,"statfs":{"total":96636764160,"available":96600174592,"internally_reserved":0,"allocated":7925760,"data_stored":58644907,"data_compressed":2871268,"data_compressed_allocated":5410816,"data_compressed_original":57345265,"omap_allocated":4394,"internal_metadata":28634838},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58916,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312924,"statfs":{"total":96636764160,"available":96576434176,"internally_reserved":0,"allocated":5619712,"data_stored":37694679,"data_compressed":2028728,"data_compressed_allocated":3444736,"data_compressed_original":36503739,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":35643695,"data_compressed":2027336,"data_compressed_allocated":3260416,"data_compressed_original":35162080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":1687856,"data_compressed":191727,"data_compressed_allocated":286720,"data_compressed_original":1635659,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2371584,"data_stored":21921646,"data_compressed":1149376,"data_compressed_allocated":1945600,"data_compressed_original":21655507,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7225344,"data_stored":64748907,"data_compressed":3518857,"data_compressed_allocated":5902336,"data_compressed_original":63871328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6557696,"data_stored":57872892,"data_compressed":3018143,"data_compressed_allocated":5267456,"data_compressed_original":57079943,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4075520,"data_stored":36462301,"data_compressed":1910825,"data_compressed_allocated":3289088,"data_compressed_original":35922107,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2170880,"data_stored":18603657,"data_compressed":964735,"data_compressed_allocated":1699840,"data_compressed_original":18332878,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6303744,"data_stored":56156186,"data_compressed":2824580,"data_compressed_allocated":5181440,"data_compressed_original":55526641,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:55:48.156 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-04T21:55:48.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:55:48.156 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-04T21:55:48.156 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-04T21:55:48.250 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:48.247+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:48.256 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:48.256 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:48.272 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:48.320 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:48.315+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:48.406 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:48.406 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:48.421 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:48.415+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:48.423 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:48.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:55:48.522 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:48.515+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-04T21:55:48.523+0000 7f1d9e303640 -1 mon.a@0(leader).osd e116 definitely_dead 0 2024-09-04T21:55:48.607 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:48.607 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:48.623 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:48.623 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:48.615+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:48.723 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:48.719+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:48.790 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:48.790 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:48.804 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:48.821 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:48.824 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:48.819+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:48.925 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:48.919+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:48.970 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:48.970 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:48.973 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:48.973 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:48.984 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:48.990 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:48.990 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:49.006 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:49.006 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:49.022 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:49.023 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:49.026 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:49.023+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:49.040 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:49.041 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:49.060 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:49.060 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:49.081 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:49.081 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:49.101 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:49.101 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:49.104 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:49.105 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:49.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:49.167 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:49.163+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:49.227 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:49.223+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:46775 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-04T21:55:49.242 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 67.878011259999994, 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023822228000000001, 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 46775 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360496+0000", 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360498+0000", 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6299999999999998e-06 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360510+0000", 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1499e-05 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360546+0000", 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5932000000000001e-05 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360568+0000", 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1984000000000002e-05 2024-09-04T21:55:49.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360712+0000", 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144182 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.360856+0000", 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144356 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0", 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.364527+0000", 2024-09-04T21:55:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036703690000000001 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384070+0000", 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.019542842000000001 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384122+0000", 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2065999999999999e-05 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:49.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384247+0000", 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012545300000000001 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:41.384318+0000", 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0914999999999994e-05 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49175 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 475136~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:49.248 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445314834999998, 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021719833000000001, 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49175 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793192+0000", 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793193+0000", 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4e-07 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793196+0000", 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793206+0000", 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4229999999999994e-06 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793213+0000", 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4370000000000001e-06 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814388+0000", 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021174454999999998 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814604+0000", 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021587900000000001 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.814912+0000", 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030848600000000001 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49290 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 40960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445284172000001, 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022433502000000001, 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.253 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49290 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793223+0000", 2024-09-04T21:55:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793224+0000", 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6599999999999996e-07 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793227+0000", 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.14e-06 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793240+0000", 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-05 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793252+0000", 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2843000000000001e-05 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815053+0000", 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021800657000000001 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815301+0000", 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024764399999999999 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815656+0000", 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035569200000000001 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49297 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 106496~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445243480000002, 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022960876000000002, 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49297 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793264+0000", 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0699999999999996e-07 2024-09-04T21:55:49.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793268+0000", 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4460000000000001e-06 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793277+0000", 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2299999999999997e-06 2024-09-04T21:55:49.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793284+0000", 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9530000000000002e-06 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815785+0000", 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022500857999999999 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.815996+0000", 2024-09-04T21:55:49.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000211065 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816224+0000", 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022851699999999999 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49303 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 172032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:49.262 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445213658, 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023578159000000001, 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49303 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793293+0000", 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793294+0000", 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793297+0000", 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9500000000000001e-06 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793306+0000", 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2140000000000002e-06 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793313+0000", 2024-09-04T21:55:49.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9800000000000001e-06 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816342+0000", 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023028835000000001 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816580+0000", 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023813 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816872+0000", 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029135000000000001 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49309 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 237568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445142695999998, 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024080300999999998, 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.267 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49309 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793364+0000", 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793365+0000", 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.054e-06 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793374+0000", 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:49.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793388+0000", 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3403e-05 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793396+0000", 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8830000000000006e-06 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.816958+0000", 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023561992 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817170+0000", 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021269799999999999 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817445+0000", 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027435799999999999 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.271 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49326 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 303104~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445067774000002, 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024582155000000001, 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.272 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49326 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793439+0000", 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793440+0000", 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.003e-06 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793445+0000", 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6840000000000004e-06 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793464+0000", 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9085000000000001e-05 2024-09-04T21:55:49.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793478+0000", 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3716999999999999e-05 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817523+0000", 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024045704000000001 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.817738+0000", 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021439899999999999 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818021+0000", 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028356300000000002 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49332 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.276 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.445015654999999, 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025083616, 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49332 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.277 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793491+0000", 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793492+0000", 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.043e-06 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793496+0000", 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793506+0000", 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9729999999999992e-06 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793516+0000", 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6269999999999995e-06 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818099+0000", 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024583880999999998 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818307+0000", 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020786200000000001 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818575+0000", 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026768400000000002 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.281 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49338 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.444981468999998, 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025617277000000001, 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49338 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793526+0000", 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5600000000000005e-07 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793533+0000", 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1930000000000004e-06 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793542+0000", 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0669999999999998e-06 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793552+0000", 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1290000000000002e-06 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818661+0000", 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025109056000000001 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.818870+0000", 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000208929 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819143+0000", 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027324700000000001 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.286 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:49344 3.a 3:550ae845:::rbd_data.11882296a3bb.0000000000000008:12 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.444945484000002, 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026178261000000001, 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 49344 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793562+0000", 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2399999999999997e-07 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793565+0000", 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8930000000000001e-06 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793574+0000", 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9129999999999992e-06 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.793581+0000", 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8399999999999997e-06 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819234+0000", 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025653441999999999 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819463+0000", 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022879500000000001 2024-09-04T21:55:49.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:45.819740+0000", 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027665399999999998 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51043 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 163840~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111408312999998, 2024-09-04T21:55:49.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022470637000000002, 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51043 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127099+0000", 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127100+0000", 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7999999999999993e-07 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127114+0000", 2024-09-04T21:55:49.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4017e-05 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127128+0000", 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4005000000000001e-05 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127139+0000", 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1007e-05 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148542+0000", 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.021403575000000001 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.148727+0000", 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018442900000000001 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149569+0000", 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084262400000000004 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.295 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51044 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 262144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111353706999999, 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023138729, 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51044 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127153+0000", 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127154+0000", 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127158+0000", 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.172e-06 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127194+0000", 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5222000000000003e-05 2024-09-04T21:55:49.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127206+0000", 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2159e-05 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149626+0000", 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.022420476000000002 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.149779+0000", 2024-09-04T21:55:49.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015296299999999999 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150292+0000", 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051277400000000002 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51045 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 360448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:49.300 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111283159000003, 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023446379, 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51045 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127224+0000", 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127225+0000", 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127230+0000", 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4110000000000004e-06 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127245+0000", 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4714e-05 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127256+0000", 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0787e-05 2024-09-04T21:55:49.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150350+0000", 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023094001999999999 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150502+0000", 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000151805 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150670+0000", 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016861 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51046 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111234832000001, 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023742831999999998, 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.305 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51046 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127272+0000", 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127273+0000", 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.23e-07 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127277+0000", 2024-09-04T21:55:49.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9650000000000002e-06 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127291+0000", 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3549e-05 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127302+0000", 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1270000000000001e-05 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150727+0000", 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023424996999999999 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.150882+0000", 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015530200000000001 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151015+0000", 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013292600000000001 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.309 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51047 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 557056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111191697999999, 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024031869000000001, 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.310 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51047 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127315+0000", 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127316+0000", 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5300000000000003e-07 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127320+0000", 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0749999999999999e-06 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127334+0000", 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3587000000000001e-05 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127344+0000", 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0461e-05 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151070+0000", 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.023725311999999998 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151219+0000", 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014918599999999999 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151347+0000", 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012839500000000001 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.314 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51048 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111149220999998, 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024346703000000001, 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51048 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127358+0000", 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127359+0000", 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.85e-07 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127375+0000", 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6084000000000001e-05 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127389+0000", 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.412e-05 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127400+0000", 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151402+0000", 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024002787000000001 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151573+0000", 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000170941 2024-09-04T21:55:49.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151705+0000", 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131161 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51049 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111079775, 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024613729000000001, 2024-09-04T21:55:49.319 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51049 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127427+0000", 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127428+0000", 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6700000000000002e-07 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127433+0000", 2024-09-04T21:55:49.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0000000000000004e-06 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127447+0000", 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4224e-05 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127459+0000", 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1112e-05 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151761+0000", 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024302397999999999 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.151914+0000", 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015300399999999999 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152041+0000", 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127024 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.323 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51050 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.111033472000003, 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024957087999999999, 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.324 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51050 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127474+0000", 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0699999999999996e-07 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127479+0000", 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7160000000000002e-06 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127494+0000", 2024-09-04T21:55:49.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4997000000000001e-05 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127506+0000", 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.168e-05 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152104+0000", 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024597810000000001 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152273+0000", 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016888100000000001 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152431+0000", 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000158097 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:51051 3.10 3:084b2591:::rbd_data.1188b450424.0000000000000006:14 [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:49.328 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.110985978000002, 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.025255452000000001, 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 51051 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.329 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127521+0000", 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127522+0000", 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5000000000000001e-07 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127527+0000", 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6299999999999997e-06 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127541+0000", 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4225e-05 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.127552+0000", 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0971999999999999e-05 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152484+0000", 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.024931803999999998 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152641+0000", 2024-09-04T21:55:49.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015691699999999999 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:46.152776+0000", 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013595400000000001 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:106971 3.0 e82/38 3:031e9e2d:::rbd_data.1188388e16dc.0000000000000000:head v 82'257, mlcod=81'254)", 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 40.608691649999997, 2024-09-04T21:55:49.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045768883000000003, 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629815+0000", 2024-09-04T21:55:49.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629817+0000", 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9400000000000001e-06 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629834+0000", 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6246999999999999e-05 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629860+0000", 2024-09-04T21:55:49.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6628999999999999e-05 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629873+0000", 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3216e-05 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.629999+0000", 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012590799999999999 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.630064+0000", 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4546000000000002e-05 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675498+0000", 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.045433922000000002 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:08.675584+0000", 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6475000000000001e-05 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.337 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.338 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.338 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:49.339 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:49.340 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:49.335+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:49.440 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:49.435+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:49.441 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:55:49.441 INFO:teuthology.orchestra.run.smithi112.stderr:marked in osd.1. 2024-09-04T21:55:49.481 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-04T21:55:49.523 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:49.524 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:49.524 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:49.524 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:49.538 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:49.541 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:49.535+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:49.643 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:49.639+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:49.702 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:49.703 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:49.703 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:49.703 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:49.703 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:49.717 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:49.744 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:49.739+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:49.841 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.223256933000002, 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.842 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:49.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:49.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:49.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.212251690000002, 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.849 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.855 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668544940000004, 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:49.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:49.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668504253999998, 2024-09-04T21:55:49.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:49.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.866 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668470854999995, 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.867 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:49.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:49.871 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668427584, 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.872 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:49.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.877 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668382902999994, 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.878 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:49.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668321746999993, 2024-09-04T21:55:49.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:49.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.889 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668291706000005, 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:49.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668246934999999, 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.895 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:49.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:49.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668198642999997, 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.904 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:49.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.909 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.668162998, 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:49.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:49.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.914 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.667913781999999, 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.915 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:49.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:49.920 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.667874913000006, 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:49.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.662728168000001, 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.926 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:49.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:49.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:49.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.931 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.662684909000006, 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.932 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:49.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.662641038000004, 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.937 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:49.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.662604759999994, 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:49.942 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:49.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.662578654000001, 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.948 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:49.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:49.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143697 3.13 3:ceb3a275:::rbd_object_map.1188388e16dc:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.4678532149999999, 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.004254991, 2024-09-04T21:55:49.953 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143697 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:49.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370766+0000", 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7429999999999998e-06 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370776+0000", 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8549999999999997e-06 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:49.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370795+0000", 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9599e-05 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370807+0000", 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1652e-05 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370896+0000", 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104000000000006e-05 2024-09-04T21:55:49.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370911+0000", 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4986e-05 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370490+0000", 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99957913600000003 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370701+0000", 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021104399999999999 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371058+0000", 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035635700000000002 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374510+0000", 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034528860000000001 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374898+0000", 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038715800000000002 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374971+0000", 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3881999999999995e-05 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375018+0000", 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6588999999999997e-05 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.961 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:49.962 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:49.962 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:49.963 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:49.964 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:49.959+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:50.064 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:50.063+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:50.090 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:50.090 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:50.090 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:50.090 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:50.105 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:50.165 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:50.159+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:50.223 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:50.224 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:50.224 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:50.224 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:50.224 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:50.238 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:50.279 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:50.275+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:50.370 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:50.363+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23, version: 95'461, data_included: [0~262144,327680~65536,507904~15827,3260416~65536,3407872~32768,3555328~65536,3670016~32768,3850240~32768], data_size: 572883, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23@95'461, size: 3883008, copy_subset: [(0, 3883008)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3883008, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.601289717, 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060935900000000003, 2024-09-04T21:55:50.416 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811327+0000", 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3629999999999999e-06 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811906+0000", 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057844000000000005 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811940+0000", 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4096e-05 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811967+0000", 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7202000000000001e-05 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812156+0000", 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189073 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812200+0000", 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3748000000000002e-05 2024-09-04T21:55:50.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.817419+0000", 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052196680000000002 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23, version: 95'467, data_included: [2883584~32768,3080192~16384,3145728~65536,3342336~49152,3440640~32768,3620864~49152], data_size: 245760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23@95'467, size: 3670016, copy_subset: [(0, 3670016)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3670016, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.583296423, 2024-09-04T21:55:50.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044989239999999996, 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829321+0000", 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5999999999999999e-06 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829583+0000", 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026244899999999999 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829622+0000", 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8359000000000003e-05 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829679+0000", 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7102999999999999e-05 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829878+0000", 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019976299999999999 2024-09-04T21:55:50.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829903+0000", 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4604999999999999e-05 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.833818+0000", 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039150449999999998 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2959 3.f e108/105 3:f7c4ed88:::rbd_data.1188388e16dc.0000000000000001:1c v 108'438, mlcod=107'435)", 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.047967698000001, 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055309019999999999, 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364649+0000", 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364676+0000", 2024-09-04T21:55:50.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7135e-05 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364688+0000", 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2082000000000001e-05 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364700+0000", 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1885999999999999e-05 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364799+0000", 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9593999999999995e-05 2024-09-04T21:55:50.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364831+0000", 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1628000000000003e-05 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370108+0000", 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052768629999999997 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370179+0000", 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1046999999999999e-05 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1379 3.a e109/108 3:573a0ada:::rbd_data.11885236e45a.00000000000000e4:1d v 109'577, mlcod=0'0)", 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:50.429 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0462326569999991, 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043781289999999997, 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366384+0000", 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1800000000000005e-07 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366392+0000", 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2439999999999993e-06 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366404+0000", 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2510000000000001e-05 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366416+0000", 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1361e-05 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368895+0000", 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024789529999999999 2024-09-04T21:55:50.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368917+0000", 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2303999999999999e-05 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370714+0000", 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017969100000000001 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370761+0000", 2024-09-04T21:55:50.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7129000000000002e-05 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1380 3.a e109/108 3:54d284e2:::rbd_data.1188e6911c8f.0000000000000046:1e v 109'579, mlcod=107'575)", 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0383831600000004, 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039209320000000002, 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374233+0000", 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-04T21:55:50.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374240+0000", 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6939999999999998e-06 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374251+0000", 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1644000000000001e-05 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374262+0000", 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0691e-05 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375369+0000", 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001106592 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375423+0000", 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4483e-05 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378103+0000", 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002680151 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378153+0000", 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9947999999999997e-05 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.438 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2965 3.13 e109/108 3:cc7f4b96:::rbd_data.1188af11ea53.000000000000001c:20 v 109'917, mlcod=109'911)", 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0292925690000008, 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039308429999999998, 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3000000000000001e-07 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383328+0000", 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6419999999999998e-06 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383337+0000", 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6270000000000006e-06 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383344+0000", 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1280000000000002e-06 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385347+0000", 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020028759999999998 2024-09-04T21:55:50.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385361+0000", 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4578e-05 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387203+0000", 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018416210000000001 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387254+0000", 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0940999999999999e-05 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1383 3.a e109/108 3:53a15561:::rbd_data.1188af11ea53.000000000000001b:20 v 109'585, mlcod=109'581)", 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.0264916520000007, 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041799580000000001, 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.443 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2699999999999999e-07 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386129+0000", 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5700000000000003e-06 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386138+0000", 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9449999999999999e-06 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386144+0000", 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3749999999999999e-06 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386257+0000", 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112911 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386305+0000", 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8087999999999998e-05 2024-09-04T21:55:50.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390256+0000", 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003950968 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390304+0000", 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7574000000000002e-05 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3023 3.11 e113/105 3:8c151454:::rbd_data.1188e5a3a295.0000000000000009:18 v 113'518, mlcod=106'512)", 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0146809780000003, 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038753149999999998, 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4e-07 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397954+0000", 2024-09-04T21:55:50.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8236e-05 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397974+0000", 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0528999999999999e-05 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397989+0000", 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4239999999999999e-05 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.399110+0000", 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00112105 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.399137+0000", 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7208000000000001e-05 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401756+0000", 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026188549999999998 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401810+0000", 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4256999999999999e-05 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1400 3.a e113/108 3:568fad06:::rbd_data.1188e5a3a295.0000000000000006:18 v 113'597, mlcod=110'594)", 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0144725499999998, 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037033109999999999, 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.452 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398144+0000", 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5300000000000004e-07 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398151+0000", 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6340000000000001e-06 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398192+0000", 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0862000000000003e-05 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398207+0000", 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.505e-05 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398405+0000", 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000197612 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398485+0000", 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0644999999999998e-05 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401799+0000", 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033140130000000002 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401846+0000", 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6842000000000003e-05 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3025 3.f e113/105 3:f69cda90:::rbd_data.1188e5a3a295.0000000000000004:18 v 113'446, mlcod=111'443)", 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0084274620000002, 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070421360000000001, 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404189+0000", 2024-09-04T21:55:50.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404195+0000", 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3149999999999997e-06 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404204+0000", 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0240000000000003e-06 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404213+0000", 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5040000000000008e-06 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404315+0000", 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010159599999999999 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404344+0000", 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9175000000000001e-05 2024-09-04T21:55:50.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411189+0000", 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006844854 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411230+0000", 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1137999999999998e-05 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.3.0:4041 3.19 e113/105 3:98fe2de1:::rbd_data.1188e5a3a295.000000000000002a:18 v 113'1164, mlcod=110'1161)", 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0083564860000003, 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007607361, 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404260+0000", 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404265+0000", 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0119999999999996e-06 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404274+0000", 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.764e-06 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404282+0000", 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1169999999999998e-06 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404538+0000", 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025666600000000001 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404564+0000", 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5415999999999999e-05 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411833+0000", 2024-09-04T21:55:50.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0072691520000000001 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411866+0000", 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3643000000000003e-05 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3026 3.f e113/105 3:f7074c4a:::rbd_data.1188e5a3a295.0000000000000024:18 v 113'448, mlcod=111'443)", 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0079610099999998, 2024-09-04T21:55:50.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073657970000000003, 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4799999999999997e-07 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404663+0000", 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404675+0000", 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2108e-05 2024-09-04T21:55:50.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404683+0000", 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2719999999999997e-06 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405007+0000", 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032386699999999998 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405035+0000", 2024-09-04T21:55:50.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7795000000000001e-05 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411899+0000", 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068636569999999996 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.412020+0000", 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121777 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144579 3.3 3:c4c92e5a:::rbd_trash:head [call rbd.trash_state_set in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.508266+0000", 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.9043497920000001, 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037309190000000001, 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144579 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.471 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508266+0000", 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508266+0000", 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508268+0000", 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2029999999999999e-06 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508278+0000", 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7240000000000004e-06 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508302+0000", 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4209000000000001e-05 2024-09-04T21:55:50.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508315+0000", 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3346000000000001e-05 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508446+0000", 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000130517 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508728+0000", 2024-09-04T21:55:50.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028195800000000001 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6", 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.509242+0000", 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051436500000000003 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.510467+0000", 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012249229999999999 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:50.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.511767+0000", 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012995789999999999 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.511886+0000", 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118954 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.511997+0000", 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000111141 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.476 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144606 3.14 3:2afd88fa:::rbd_data.1188c8e6983d.0000000000000000:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.873926242, 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010126151, 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144606 2024-09-04T21:55:50.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538692+0000", 2024-09-04T21:55:50.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2340000000000001e-06 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538701+0000", 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7829999999999993e-06 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538755+0000", 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3352e-05 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538770+0000", 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5027e-05 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538877+0000", 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010716 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539557+0000", 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067977599999999999 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539848+0000", 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000291253 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544640+0000", 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047919900000000003 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548544+0000", 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039044190000000001 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548701+0000", 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015645599999999999 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548815+0000", 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114701 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.482 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144607 3.1c 3:3e16db79:::rbd_data.1188c8e6983d.0000000000000001:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8738156740000003, 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070450139999999996, 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:50.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144607 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538801+0000", 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0070000000000001e-06 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538806+0000", 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.746e-06 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538819+0000", 2024-09-04T21:55:50.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2894000000000001e-05 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538827+0000", 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1820000000000006e-06 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538963+0000", 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135981 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539501+0000", 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053799499999999999 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539807+0000", 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030629000000000002 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544657+0000", 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048502149999999997 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545514+0000", 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085667800000000004 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545676+0000", 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016225300000000001 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545845+0000", 2024-09-04T21:55:50.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016877299999999999 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144613 3.1c 3:38cae06c:::rbd_data.1188c8e6983d.0000000000000013:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8737264489999998, 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0099469250000000006, 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:50.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144613 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:50.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538898+0000", 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538906+0000", 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3659999999999992e-06 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538930+0000", 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4079000000000001e-05 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538952+0000", 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1437000000000002e-05 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542268+0000", 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033164679999999999 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542700+0000", 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043201699999999998 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543018+0000", 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031757600000000002 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545938+0000", 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002920387 2024-09-04T21:55:50.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548603+0000", 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026646299999999999 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548736+0000", 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013379 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548836+0000", 2024-09-04T21:55:50.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9535000000000002e-05 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:144608 3.b e114/105 3:d125cd18:::rbd_data.1188c8e6983d.0000000000000002:head v 114'334, mlcod=106'332)", 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8704207950000002, 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005416677, 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542197+0000", 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3870000000000002e-06 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542213+0000", 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6362999999999999e-05 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542243+0000", 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9833e-05 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542260+0000", 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6903000000000001e-05 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543925+0000", 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001664741 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543988+0000", 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3335000000000003e-05 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547491+0000", 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035024880000000002 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547611+0000", 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120627 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.499 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144621 3.1c 3:3a3265fe:::rbd_data.1188c8e6983d.000000000000002e:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.866463403, 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004689365, 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.500 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144621 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546156+0000", 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.523e-06 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546172+0000", 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6353000000000001e-05 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546208+0000", 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6269000000000001e-05 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546230+0000", 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1849999999999999e-05 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546385+0000", 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015487699999999999 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546843+0000", 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045768300000000002 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547462+0000", 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061928999999999999 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.549650+0000", 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021879289999999999 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550617+0000", 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096704600000000005 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550743+0000", 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012623300000000001 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550841+0000", 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8312000000000003e-05 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144622 3.6 3:60fdb67d:::rbd_data.1188c8e6983d.000000000000002f:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.547104+0000", 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8655116970000001, 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042288610000000004, 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.506 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144622 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547104+0000", 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547104+0000", 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547114+0000", 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7229999999999999e-06 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547124+0000", 2024-09-04T21:55:50.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0509999999999999e-05 2024-09-04T21:55:50.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547152+0000", 2024-09-04T21:55:50.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8422e-05 2024-09-04T21:55:50.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547172+0000", 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9106999999999998e-05 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547321+0000", 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014964000000000001 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548478+0000", 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001156628 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548779+0000", 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030094399999999998 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550726+0000", 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019473540000000001 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.551181+0000", 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045461399999999999 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.551266+0000", 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5490999999999999e-05 2024-09-04T21:55:50.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.551333+0000", 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6427999999999995e-05 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3099 3.11 e117/105 3:8f806b3e:::rbd_data.1188b450424.000000000000004a:14 v 117'523, mlcod=115'520)", 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:49.450435+0000", 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.96218063700000001, 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003747458, 2024-09-04T21:55:50.515 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450435+0000", 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450435+0000", 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450436+0000", 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.037e-06 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450445+0000", 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6999999999999997e-06 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:50.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450458+0000", 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3023999999999999e-05 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450467+0000", 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2380000000000003e-06 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450566+0000", 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8776000000000004e-05 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450593+0000", 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7273000000000001e-05 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.454083+0000", 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034895690000000001 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.454182+0000", 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9840999999999995e-05 2024-09-04T21:55:50.519 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.520 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:50.520 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:50.520 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:50.521 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:50.522 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:50.519+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:50.621 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:50.615+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:50.650 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:50.651 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:50.651 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:50.651 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:50.665 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:50.722 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:50.719+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:50.788 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:50.788 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:50.788 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:50.802 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:50.823 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:50.819+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:50.924 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:50.923+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:50.971 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29485719900001, 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:50.972 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:50.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:50.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294830414, 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:50.977 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:50.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.981 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29480148099999, 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:50.982 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:50.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294777834, 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294758053, 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294738658, 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29471938499999, 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.006 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294700163, 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:51.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294681326, 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29466192300001, 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294640867, 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:51.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294608689, 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.026 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:51.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:51.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294582468, 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.031 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:51.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.035 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29455425899999, 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.036 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:51.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294534219, 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.041 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:51.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29451483699999, 2024-09-04T21:55:51.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:51.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:51.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294491249, 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:51.050 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:51.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:51.054 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294470932, 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.055 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:51.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.059 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.29445138, 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:51.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:51.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:51.064 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.294431621, 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:51.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:51.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:51.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:51.069 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:51.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:51.071 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:51.067+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:51.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:51.171 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:51.167+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:51.241 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:51.242 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:51.242 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:51.242 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:51.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:51.302 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:51.299+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:51.307 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:51.308 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:51.324 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:51.325 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:51.341 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:51.341 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:51.360 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:51.360 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:51.373 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:51.367+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:51.393 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:51.393 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:51.399 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:51.399 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:51.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:51.425 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:51.425 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:51.449 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:51.449 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:51.473 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:51.467+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:51.478 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:51.478 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:51.482 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:55:51.567 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:51.567 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:51.574 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:51.571+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:51.584 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:51.675 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:51.671+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:51.756 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:51.756 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:51.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:51.814 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:51.811+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:51.877 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:51.871+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:51.969 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:51.969 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:51.977 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:51.971+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:51.983 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:52.077 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:52.071+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:52.100 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:52.100 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:52.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:52.178 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:52.171+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:52.238 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:52.238 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:52.251 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:52.294 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:52.291+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:52.371 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:52.371 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:52.380 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:52.375+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:52.385 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:52.457 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:55:52.480 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:52.475+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:52.561 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:52.561 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:52.577 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:52.624 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:52.619+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:52.681 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:52.675+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:52.708 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:52.708 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:52.722 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:52.815 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:52.811+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:52.883 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:52.879+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:52.893 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:52.893 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:52.907 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:53.000 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:52.995+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:53.077 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:53.077 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:53.086 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:53.079+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:53.091 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:53.186 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:53.179+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:53.258 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:53.258 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:53.272 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:53.315 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:53.311+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:53.388 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:53.383+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:53.438 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:53.438 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:53.453 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:53.499 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:53.495+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:53.503 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:53.581 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:53.582 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:53.589 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:53.583+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:53.596 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:53.653 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:53.654 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:53.669 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:53.670 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:53.686 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:53.686 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:53.690 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:53.687+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:53.702 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:53.720 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:53.721 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:53.741 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:53.741 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:53.762 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:53.762 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:53.765 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:53.765 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:53.781 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:53.782 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:53.782 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:53.791 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:53.787+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:53.892 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:53.887+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:53.899 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:53.899 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:53.899 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:53.899 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:53.899 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:53.913 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:53.999 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:53.995+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.463686396, 2024-09-04T21:55:54.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:54.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:54.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.452681153, 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:54.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908974403000002, 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:54.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.097 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908933716999996, 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:54.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:54.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908900317999993, 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:54.102 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:54.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:54.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908857046999998, 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:54.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:54.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908812366000006, 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.111 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:54.115 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908751210000005, 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.116 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:54.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:54.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908721169000003, 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.121 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:54.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:54.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.125 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908676397999997, 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:54.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:54.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:54.130 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908628105999995, 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:54.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:54.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908592460999998, 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.135 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:54.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.139 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908343244999998, 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:54.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.908304376000004, 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.144 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:54.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:54.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.148 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.903157630999999, 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:54.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:54.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:54.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.903114372000005, 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:54.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:54.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.903070501000002, 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:54.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:54.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.903034223000006, 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:54.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.903008116999999, 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:54.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:54.170 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143697 3.13 3:ceb3a275:::rbd_object_map.1188388e16dc:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.708282678, 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.004254991, 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.171 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143697 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370766+0000", 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7429999999999998e-06 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370776+0000", 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8549999999999997e-06 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370795+0000", 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9599e-05 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370807+0000", 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1652e-05 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370896+0000", 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104000000000006e-05 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370911+0000", 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4986e-05 2024-09-04T21:55:54.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370490+0000", 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99957913600000003 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370701+0000", 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021104399999999999 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:54.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371058+0000", 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035635700000000002 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374510+0000", 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034528860000000001 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374898+0000", 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038715800000000002 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374971+0000", 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3881999999999995e-05 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375018+0000", 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6588999999999997e-05 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.177 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:54.179 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:54.179 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:54.171+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:54.279 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:54.275+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:54.350 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:54.351 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:54.351 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:54.351 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:54.365 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:54.380 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:54.374+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:54.482 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:54.475+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:54.482 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-04T21:55:54.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:55:54.482 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-04T21:55:54.482 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-04T21:55:54.482 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-04T21:55:54.483 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-09-04T21:55:54.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:55:54.544 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:54.544 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:54.544 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:54.544 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:54.545 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:54.563 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:54.611 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:54.607+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:54.616 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:54.616 INFO:teuthology.orchestra.run.smithi157.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-04T21:55:54.616 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:54.683 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:54.679+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:54.689 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:54.689 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:54.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23, version: 95'461, data_included: [0~262144,327680~65536,507904~15827,3260416~65536,3407872~32768,3555328~65536,3670016~32768,3850240~32768], data_size: 572883, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23@95'461, size: 3883008, copy_subset: [(0, 3883008)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3883008, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.875411763000001, 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060935900000000003, 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811327+0000", 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3629999999999999e-06 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811906+0000", 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057844000000000005 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811940+0000", 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4096e-05 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811967+0000", 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7202000000000001e-05 2024-09-04T21:55:54.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812156+0000", 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189073 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812200+0000", 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3748000000000002e-05 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.817419+0000", 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052196680000000002 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23, version: 95'467, data_included: [2883584~32768,3080192~16384,3145728~65536,3342336~49152,3440640~32768,3620864~49152], data_size: 245760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:be9c54c2:::rbd_data.1188f71f571e.000000000000000e:23@95'467, size: 3670016, copy_subset: [(0, 3670016)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3670016, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.857418469000001, 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044989239999999996, 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.694 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829319+0000", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829321+0000", 2024-09-04T21:55:54.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5999999999999999e-06 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829583+0000", 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026244899999999999 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829622+0000", 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8359000000000003e-05 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829679+0000", 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7102999999999999e-05 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829878+0000", 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019976299999999999 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.829903+0000", 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4604999999999999e-05 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.833818+0000", 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039150449999999998 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2959 3.f e108/105 3:f7c4ed88:::rbd_data.1188388e16dc.0000000000000001:1c v 108'438, mlcod=107'435)", 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.322089743999999, 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055309019999999999, 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.698 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364649+0000", 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364676+0000", 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7135e-05 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364688+0000", 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2082000000000001e-05 2024-09-04T21:55:54.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364700+0000", 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1885999999999999e-05 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364799+0000", 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9593999999999995e-05 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364831+0000", 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1628000000000003e-05 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370108+0000", 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052768629999999997 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370179+0000", 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1046999999999999e-05 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.702 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1379 3.a e109/108 3:573a0ada:::rbd_data.11885236e45a.00000000000000e4:1d v 109'577, mlcod=0'0)", 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.320354703, 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0043781289999999997, 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366383+0000", 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366384+0000", 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1800000000000005e-07 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366392+0000", 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2439999999999993e-06 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366404+0000", 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2510000000000001e-05 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.366416+0000", 2024-09-04T21:55:54.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1361e-05 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368895+0000", 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024789529999999999 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.368917+0000", 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2303999999999999e-05 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370714+0000", 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017969100000000001 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370761+0000", 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7129000000000002e-05 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1380 3.a e109/108 3:54d284e2:::rbd_data.1188e6911c8f.0000000000000046:1e v 109'579, mlcod=107'575)", 2024-09-04T21:55:54.707 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.312505206000001, 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039209320000000002, 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374232+0000", 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374233+0000", 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374240+0000", 2024-09-04T21:55:54.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6939999999999998e-06 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374251+0000", 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1644000000000001e-05 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374262+0000", 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0691e-05 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375369+0000", 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001106592 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375423+0000", 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4483e-05 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378103+0000", 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002680151 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.378153+0000", 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9947999999999997e-05 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2965 3.13 e109/108 3:cc7f4b96:::rbd_data.1188af11ea53.000000000000001c:20 v 109'917, mlcod=109'911)", 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.303414614999999, 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039308429999999998, 2024-09-04T21:55:54.712 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383323+0000", 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3000000000000001e-07 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383328+0000", 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6419999999999998e-06 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383337+0000", 2024-09-04T21:55:54.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6270000000000006e-06 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.383344+0000", 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1280000000000002e-06 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385347+0000", 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020028759999999998 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.385361+0000", 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4578e-05 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387203+0000", 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018416210000000001 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.387254+0000", 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0940999999999999e-05 2024-09-04T21:55:54.716 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1383 3.a e109/108 3:53a15561:::rbd_data.1188af11ea53.000000000000001b:20 v 109'585, mlcod=109'581)", 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.300613697999999, 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041799580000000001, 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386124+0000", 2024-09-04T21:55:54.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2699999999999999e-07 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386129+0000", 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5700000000000003e-06 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386138+0000", 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9449999999999999e-06 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386144+0000", 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3749999999999999e-06 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386257+0000", 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112911 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.386305+0000", 2024-09-04T21:55:54.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8087999999999998e-05 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390256+0000", 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003950968 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.390304+0000", 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7574000000000002e-05 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3023 3.11 e113/105 3:8c151454:::rbd_data.1188e5a3a295.0000000000000009:18 v 113'518, mlcod=106'512)", 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2888030239999999, 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038753149999999998, 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397935+0000", 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4e-07 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397954+0000", 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8236e-05 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397974+0000", 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0528999999999999e-05 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.397989+0000", 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4239999999999999e-05 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.399110+0000", 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00112105 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.399137+0000", 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7208000000000001e-05 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401756+0000", 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026188549999999998 2024-09-04T21:55:54.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401810+0000", 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4256999999999999e-05 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.6.0:1400 3.a e113/108 3:568fad06:::rbd_data.1188e5a3a295.0000000000000006:18 v 113'597, mlcod=110'594)", 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2885945959999994, 2024-09-04T21:55:54.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037033109999999999, 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398143+0000", 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398144+0000", 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5300000000000004e-07 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398151+0000", 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6340000000000001e-06 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398192+0000", 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0862000000000003e-05 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398207+0000", 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.505e-05 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398405+0000", 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000197612 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.398485+0000", 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0644999999999998e-05 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401799+0000", 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033140130000000002 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.401846+0000", 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6842000000000003e-05 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.730 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3025 3.f e113/105 3:f69cda90:::rbd_data.1188e5a3a295.0000000000000004:18 v 113'446, mlcod=111'443)", 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2825495080000007, 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070421360000000001, 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404189+0000", 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-04T21:55:54.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404195+0000", 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3149999999999997e-06 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404204+0000", 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0240000000000003e-06 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404213+0000", 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5040000000000008e-06 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404315+0000", 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010159599999999999 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404344+0000", 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9175000000000001e-05 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411189+0000", 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006844854 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411230+0000", 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1137999999999998e-05 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.3.0:4041 3.19 e113/105 3:98fe2de1:::rbd_data.1188e5a3a295.000000000000002a:18 v 113'1164, mlcod=110'1161)", 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2824785320000007, 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007607361, 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404260+0000", 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404265+0000", 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0119999999999996e-06 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404274+0000", 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.764e-06 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404282+0000", 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1169999999999998e-06 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404538+0000", 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025666600000000001 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404564+0000", 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5415999999999999e-05 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411833+0000", 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0072691520000000001 2024-09-04T21:55:54.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411866+0000", 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3643000000000003e-05 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3026 3.f e113/105 3:f7074c4a:::rbd_data.1188e5a3a295.0000000000000024:18 v 113'448, mlcod=111'443)", 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2820830559999994, 2024-09-04T21:55:54.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073657970000000003, 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4799999999999997e-07 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404663+0000", 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404675+0000", 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2108e-05 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404683+0000", 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2719999999999997e-06 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405007+0000", 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032386699999999998 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405035+0000", 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7795000000000001e-05 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411899+0000", 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068636569999999996 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.412020+0000", 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121777 2024-09-04T21:55:54.744 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144579 3.3 3:c4c92e5a:::rbd_trash:head [call rbd.trash_state_set in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.508266+0000", 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1784718380000001, 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037309190000000001, 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144579 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508266+0000", 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508266+0000", 2024-09-04T21:55:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508268+0000", 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2029999999999999e-06 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508278+0000", 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7240000000000004e-06 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508302+0000", 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4209000000000001e-05 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508315+0000", 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3346000000000001e-05 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508446+0000", 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000130517 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.508728+0000", 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028195800000000001 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6", 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.509242+0000", 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051436500000000003 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.510467+0000", 2024-09-04T21:55:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012249229999999999 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.511767+0000", 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012995789999999999 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.511886+0000", 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118954 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.511997+0000", 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000111141 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144606 3.14 3:2afd88fa:::rbd_data.1188c8e6983d.0000000000000000:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.148048288, 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010126151, 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144606 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538692+0000", 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2340000000000001e-06 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538701+0000", 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7829999999999993e-06 2024-09-04T21:55:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538755+0000", 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3352e-05 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538770+0000", 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5027e-05 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538877+0000", 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010716 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539557+0000", 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067977599999999999 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539848+0000", 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000291253 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544640+0000", 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047919900000000003 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548544+0000", 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039044190000000001 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548701+0000", 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015645599999999999 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548815+0000", 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114701 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144607 3.1c 3:3e16db79:::rbd_data.1188c8e6983d.0000000000000001:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1479377199999998, 2024-09-04T21:55:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070450139999999996, 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144607 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538801+0000", 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0070000000000001e-06 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538806+0000", 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.746e-06 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538819+0000", 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2894000000000001e-05 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538827+0000", 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1820000000000006e-06 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538963+0000", 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135981 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539501+0000", 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053799499999999999 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539807+0000", 2024-09-04T21:55:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030629000000000002 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544657+0000", 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048502149999999997 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545514+0000", 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085667800000000004 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545676+0000", 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016225300000000001 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545845+0000", 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016877299999999999 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144613 3.1c 3:38cae06c:::rbd_data.1188c8e6983d.0000000000000013:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1478484949999999, 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0099469250000000006, 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144613 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538898+0000", 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538906+0000", 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3659999999999992e-06 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538930+0000", 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4079000000000001e-05 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538952+0000", 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1437000000000002e-05 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542268+0000", 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033164679999999999 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542700+0000", 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043201699999999998 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543018+0000", 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031757600000000002 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545938+0000", 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002920387 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548603+0000", 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026646299999999999 2024-09-04T21:55:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548736+0000", 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013379 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548836+0000", 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9535000000000002e-05 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:144608 3.b e114/105 3:d125cd18:::rbd_data.1188c8e6983d.0000000000000002:head v 114'334, mlcod=106'332)", 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1445428409999998, 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005416677, 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542197+0000", 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3870000000000002e-06 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542213+0000", 2024-09-04T21:55:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6362999999999999e-05 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542243+0000", 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9833e-05 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542260+0000", 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6903000000000001e-05 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543925+0000", 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001664741 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543988+0000", 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3335000000000003e-05 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547491+0000", 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035024880000000002 2024-09-04T21:55:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547611+0000", 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120627 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144621 3.1c 3:3a3265fe:::rbd_data.1188c8e6983d.000000000000002e:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1405854489999996, 2024-09-04T21:55:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004689365, 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144621 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546156+0000", 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.523e-06 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546172+0000", 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6353000000000001e-05 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546208+0000", 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6269000000000001e-05 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546230+0000", 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1849999999999999e-05 2024-09-04T21:55:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546385+0000", 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015487699999999999 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546843+0000", 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045768300000000002 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547462+0000", 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061928999999999999 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.549650+0000", 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021879289999999999 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550617+0000", 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096704600000000005 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550743+0000", 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012623300000000001 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550841+0000", 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8312000000000003e-05 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144622 3.6 3:60fdb67d:::rbd_data.1188c8e6983d.000000000000002f:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.547104+0000", 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1396337429999992, 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0042288610000000004, 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144622 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547104+0000", 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547104+0000", 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547114+0000", 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7229999999999999e-06 2024-09-04T21:55:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547124+0000", 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0509999999999999e-05 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547152+0000", 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8422e-05 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547172+0000", 2024-09-04T21:55:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9106999999999998e-05 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547321+0000", 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014964000000000001 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548478+0000", 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001156628 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548779+0000", 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030094399999999998 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550726+0000", 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019473540000000001 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.551181+0000", 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045461399999999999 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.551266+0000", 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5490999999999999e-05 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.551333+0000", 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6427999999999995e-05 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3099 3.11 e117/105 3:8f806b3e:::rbd_data.1188b450424.000000000000004a:14 v 117'523, mlcod=115'520)", 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:49.450435+0000", 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.2363026829999999, 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003747458, 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450435+0000", 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450435+0000", 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450436+0000", 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.037e-06 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450445+0000", 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6999999999999997e-06 2024-09-04T21:55:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450458+0000", 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3023999999999999e-05 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450467+0000", 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2380000000000003e-06 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450566+0000", 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8776000000000004e-05 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.450593+0000", 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7273000000000001e-05 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.454083+0000", 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034895690000000001 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:49.454182+0000", 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9840999999999995e-05 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:54.791 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:54.793 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:54.787+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:54.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:54.894 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:54.890+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:54.913 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:54.913 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:54.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:55.014 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:55.010+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:55.091 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:55.091 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:55.091 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:55.091 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:55.091 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:55.095 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:55.090+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:55.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:55.196 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:55.190+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:55.267 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.267 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58863768000001, 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:55.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:55.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.272 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588610895, 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.273 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:55.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58858196200001, 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:55.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:55.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588558315, 2024-09-04T21:55:55.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:55.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58853853399999, 2024-09-04T21:55:55.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:55.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588519139, 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.292 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:55.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58849986600001, 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.297 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:55.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588480644, 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:55.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588461807, 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:55.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:55.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58844240400001, 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.311 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588421348, 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.316 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:55.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:55.320 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58838917, 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:55.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588362949, 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:55.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58833473999999, 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:55.329 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:55.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.333 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.5883147, 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:55.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58829531799999, 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:55.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58827173, 2024-09-04T21:55:55.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:55.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:55.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.351 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.58825141299999, 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.352 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:55.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588231861, 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:55.358 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.359 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:55.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:55.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.588212102, 2024-09-04T21:55:55.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.366 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.372 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:55.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:55:55.375 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:55.367+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:55.474 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:55.467+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:55.554 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:145616 3.4 3:2753a72f:::rbd_data.11882296a3bb.0000000000000009:head [delete] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.553616+0000", 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.00062350600000000002, 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070494700000000004, 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:55.555 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 145616 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553616+0000", 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553616+0000", 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553618+0000", 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3360000000000002e-06 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553628+0000", 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8369999999999992e-06 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553660+0000", 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1699000000000003e-05 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553681+0000", 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1739e-05 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.553863+0000", 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018133799999999999 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.554247+0000", 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038462300000000001 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 1 2024-09-04T21:55:55.559 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:55.572 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:55:55.575 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:55.570+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:55.675 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:55.670+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:55.699 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:55.699 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:55.700 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:55.700 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:55.700 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:55.714 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:55:55.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:55:55.802 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:55.798+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:55.877 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:55.870+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:55.883 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.029493452, 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:55:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:55:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:55:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:55:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.90344618899999, 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:55:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:55:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:55:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1516 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [set-alloc-hint object_size 524288 write_size 524288,write 426508~31517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.706892551, 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036634910000000001, 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1516 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174323+0000", 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174327+0000", 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1559999999999997e-06 2024-09-04T21:55:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174401+0000", 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4532999999999999e-05 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174443+0000", 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2049999999999999e-05 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174463+0000", 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0083000000000001e-05 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174643+0000", 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179068 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.174898+0000", 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025522099999999997 2024-09-04T21:55:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.175146+0000", 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024854199999999998 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.176804+0000", 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016580150000000001 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177821+0000", 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010166350000000001 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177911+0000", 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9802000000000005e-05 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.177986+0000", 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5386e-05 2024-09-04T21:55:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.656238582, 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:55:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:55:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.628480792, 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:55:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.62836523, 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 109.169057893, 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:55:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:55:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 109.16883183500001, 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:55:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.69161876699999, 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:55:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:55:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.462230656, 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:55:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:55:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:55:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7607 3.19 3:9e7f448d:::rbd_data.1188b1294ada.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.164040015, 2024-09-04T21:55:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038231469999999998, 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7607 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717175+0000", 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717178+0000", 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2770000000000001e-06 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717190+0000", 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1735e-05 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717226+0000", 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5966000000000001e-05 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717244+0000", 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7354999999999999e-05 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717418+0000", 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017423699999999999 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717565+0000", 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014774100000000001 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717733+0000", 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016791399999999999 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719953+0000", 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022200380000000001 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720831+0000", 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087719299999999996 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720922+0000", 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1377000000000003e-05 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720998+0000", 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6314000000000002e-05 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.16392716, 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:55:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 107.984187781, 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:55:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:55:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 107.317860876, 2024-09-04T21:55:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:55:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:55:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:55:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4488.0:7880 3.1a e32/22 3:590d5006:::rbd_data.1188b1294ada.0000000000000006:head v 32'36, mlcod=30'33)", 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 107.2964107, 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0036093789999999998, 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584804+0000", 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584805+0000", 2024-09-04T21:55:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0999999999999998e-07 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584820+0000", 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4866e-05 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584862+0000", 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2058000000000003e-05 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.584880+0000", 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7648000000000001e-05 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585051+0000", 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017147199999999999 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.585194+0000", 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014230800000000001 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588347+0000", 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031538999999999998 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.588414+0000", 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6416999999999998e-05 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:145127 3.19 3.1e5da059 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:55:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.4227148960000005, 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90329116399999998, 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 145127 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:55:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458503+0000", 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0029999999999999e-06 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458512+0000", 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3590000000000008e-06 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458530+0000", 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8790000000000001e-05 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458539+0000", 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2290000000000003e-06 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458630+0000", 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0841000000000007e-05 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458644+0000", 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361754+0000", 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90310956399999998 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361791+0000", 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7728999999999997e-05 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4160 3.8 e122/121 3:1505c1f6:::rbd_data.1188b450424.0000000000000003:14 v 122'534, mlcod=0'0)", 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.3890348239999999, 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0042635119999999997, 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492182+0000", 2024-09-04T21:55:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492200+0000", 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8009e-05 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492229+0000", 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8469000000000001e-05 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492249+0000", 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9884000000000001e-05 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492350+0000", 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000101572 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492378+0000", 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8138999999999999e-05 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496377+0000", 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0039985480000000002 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496444+0000", 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6828000000000005e-05 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4161 3.8 e122/121 3:13882358:::rbd_data.1188b450424.0000000000000004:14 v 122'536, mlcod=0'0)", 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.386250771, 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057478820000000002, 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494967+0000", 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494982+0000", 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5226e-05 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495010+0000", 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8143000000000001e-05 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495029+0000", 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8872000000000002e-05 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495238+0000", 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020955799999999999 2024-09-04T21:55:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495292+0000", 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3548999999999998e-05 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500587+0000", 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052953439999999996 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500712+0000", 2024-09-04T21:55:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012510000000000001 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4171 3.8 e123/121 3:16896659:::rbd_data.11882296a3bb.00000000000001d8:12 v 123'538, mlcod=122'534)", 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.51811797699999995, 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053660619999999996, 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363098+0000", 2024-09-04T21:55:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363103+0000", 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9269999999999996e-06 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363114+0000", 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1077000000000001e-05 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363137+0000", 2024-09-04T21:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363420+0000", 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028287199999999998 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363445+0000", 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5471999999999999e-05 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368381+0000", 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0049356000000000001 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368463+0000", 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2495000000000002e-05 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:276 3.1a e123/120 3:58e50b11:::rbd_data.11882296a3bb.000000000000004e:12 v 123'923, mlcod=116'919)", 2024-09-04T21:55:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.51633191300000003, 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059945689999999999, 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364884+0000", 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9599999999999999e-07 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364893+0000", 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6000000000000007e-06 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364919+0000", 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6156000000000002e-05 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364936+0000", 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6920000000000001e-05 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365020+0000", 2024-09-04T21:55:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4146000000000004e-05 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365036+0000", 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6277e-05 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370836+0000", 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058004290000000002 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370878+0000", 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1344999999999998e-05 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:55.998 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:56.000 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:55:56.000 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:56.000 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:56.001 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:56.001 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:56.001 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:56.001 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:56.002 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:56.002 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:56.002 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:55.994+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:56.015 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:56.015 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:56.036 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:56.036 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:56.057 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:56.057 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:56.078 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:56.078 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:56.100 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:56.095+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:56.130 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:56.130 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:56.146 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:55:56.201 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:56.199+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:56.302 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:56.298+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:56.315 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:56.315 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:56.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:55:56.426 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:56.422+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:56.504 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:56.499+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:56.510 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:56.510 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:56.526 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:55:56.618 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:56.614+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:56.700 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:56.700 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:56.706 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:56.702+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:56.717 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:55:56.807 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:56.802+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:56.886 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:56.886 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:56.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:55:56.907 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:56.903+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:57.008 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:57.002+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:57.072 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:57.072 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:57.086 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:55:57.130 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:57.126+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:57.209 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:57.206+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:57.218 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:57.218 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:57.233 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:55:57.310 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:57.303+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:57.356 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:57.357 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:57.372 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:55:57.390 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:55:57.415 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:57.411+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:57.512 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:57.507+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:57.550 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:57.550 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:57.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:55:57.651 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:57.647+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:57.714 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:57.707+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:57.732 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:57.732 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:57.746 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:55:57.831 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:57.827+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:57.909 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:57.909 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:57.916 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:55:57.911+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:55:57.924 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:55:58.017 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:58.010+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:58.092 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:58.093 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:58.097 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:55:58.108 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:55:58.142 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:58.138+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:58.220 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:58.214+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:58.225 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:58.225 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:58.239 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:55:58.253 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:55:58.254 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:55:58.270 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:55:58.270 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:55:58.286 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:55:58.286 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:55:58.303 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:55:58.304 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:55:58.320 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:58.314+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:58.321 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:55:58.322 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:55:58.341 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:55:58.342 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:55:58.362 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:55:58.363 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:55:58.366 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:55:58.366 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:55:58.380 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:55:58.383 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:55:58.383 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:55:58.427 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:58.423+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33065 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:58.501 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 87.882520231000001, 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96034945900000002, 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33065 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615359+0000", 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615361+0000", 2024-09-04T21:55:58.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9099999999999999e-06 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615369+0000", 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.559e-06 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615379+0000", 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0336999999999999e-05 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615391+0000", 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1538e-05 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615491+0000", 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010035399999999999 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.615498+0000", 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3799999999999996e-06 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575689+0000", 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.96019046500000005 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575709+0000", 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9916e-05 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:33066 3.11 3.5fc03b71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 87.871514988000001, 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94949143300000005, 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.507 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 33066 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626365+0000", 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626368+0000", 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6399999999999999e-06 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626384+0000", 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5438999999999999e-05 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626423+0000", 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9112000000000001e-05 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626448+0000", 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5602999999999999e-05 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626653+0000", 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020436299999999999 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:30.626670+0000", 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7696000000000001e-05 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575827+0000", 2024-09-04T21:55:58.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.94915677499999995 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:31.575856+0000", 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8804999999999999e-05 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36765 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:58.512 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327808238000003, 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066973424000000004, 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36765 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:58.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170071+0000", 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170072+0000", 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.61e-07 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170083+0000", 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0932e-05 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170090+0000", 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2309999999999999e-06 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170096+0000", 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7540000000000003e-06 2024-09-04T21:55:58.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236858+0000", 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066762029000000001 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.236899+0000", 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0893000000000003e-05 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237045+0000", 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014592399999999999 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36766 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327767551999997, 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067220574000000005, 2024-09-04T21:55:58.517 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36766 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170112+0000", 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170113+0000", 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2900000000000003e-07 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170115+0000", 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4660000000000002e-06 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170122+0000", 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0729999999999999e-06 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170137+0000", 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5124e-05 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237074+0000", 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066936443999999998 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237131+0000", 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7349999999999998e-05 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237333+0000", 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020148799999999999 2024-09-04T21:55:58.521 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36767 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327734152999994, 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067400126000000005, 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.522 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36767 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170145+0000", 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170146+0000", 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2600000000000002e-07 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170162+0000", 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5667000000000001e-05 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170173+0000", 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.146e-05 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170179+0000", 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.857e-06 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237357+0000", 2024-09-04T21:55:58.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067177723999999994 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237397+0000", 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0163999999999997e-05 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237546+0000", 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014872799999999999 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.526 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36768 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327690881999999, 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067681187000000004, 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.527 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36768 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170189+0000", 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3799999999999997e-07 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170193+0000", 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5219999999999999e-06 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170217+0000", 2024-09-04T21:55:58.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3947e-05 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170223+0000", 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4640000000000003e-06 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237577+0000", 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067354147000000003 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237637+0000", 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9363000000000001e-05 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237870+0000", 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023310599999999999 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36769 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327646200999993, 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.06788777, 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36769 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.532 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170233+0000", 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170234+0000", 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2300000000000001e-07 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170237+0000", 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6290000000000002e-06 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170254+0000", 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6994e-05 2024-09-04T21:55:58.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170265+0000", 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1612e-05 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237897+0000", 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067631992000000002 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.237940+0000", 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3044999999999999e-05 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238121+0000", 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018087499999999999 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.536 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36770 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327585045000006, 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068149708000000003, 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36770 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170295+0000", 2024-09-04T21:55:58.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8999999999999995e-07 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170298+0000", 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170308+0000", 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3919999999999992e-06 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170315+0000", 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6830000000000008e-06 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238150+0000", 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067835018999999996 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238211+0000", 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1208000000000001e-05 2024-09-04T21:55:58.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238444+0000", 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000232737 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36771 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327555004000004, 2024-09-04T21:55:58.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068438371999999997, 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36771 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:58.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170325+0000", 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1200000000000003e-07 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170327+0000", 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2129999999999998e-06 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170345+0000", 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7473e-05 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170351+0000", 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1310000000000003e-06 2024-09-04T21:55:58.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238469+0000", 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068117662999999995 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238526+0000", 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7263999999999999e-05 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238763+0000", 2024-09-04T21:55:58.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000237116 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36772 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327510232999998, 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068667169, 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36772 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170369+0000", 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170370+0000", 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0299999999999998e-07 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170373+0000", 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.909e-06 2024-09-04T21:55:58.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170393+0000", 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0469999999999999e-05 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170400+0000", 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5540000000000003e-06 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238788+0000", 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068387873000000002 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.238844+0000", 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6434000000000002e-05 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239036+0000", 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019222600000000001 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36773 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327461940999996, 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068871355999999995, 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.551 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36773 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170418+0000", 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1900000000000002e-07 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170421+0000", 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.038e-06 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170429+0000", 2024-09-04T21:55:58.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5530000000000004e-06 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170444+0000", 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4577e-05 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239072+0000", 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068628103999999995 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239123+0000", 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1609e-05 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239289+0000", 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000165756 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.555 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36774 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 761856~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327426295999999, 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069129109999999994, 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36774 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170453+0000", 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170653+0000", 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020013999999999999 2024-09-04T21:55:58.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170658+0000", 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.211e-06 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170669+0000", 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1065e-05 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170690+0000", 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1189999999999999e-05 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239321+0000", 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068631302000000005 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239378+0000", 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6813e-05 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239582+0000", 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000204389 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36775 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327177079999998, 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069100756999999999, 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.560 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36775 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170702+0000", 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170703+0000", 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4899999999999995e-07 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170706+0000", 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.892e-06 2024-09-04T21:55:58.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170716+0000", 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0032e-05 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170732+0000", 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5756e-05 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239607+0000", 2024-09-04T21:55:58.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.068875127999999994 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239650+0000", 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3337999999999997e-05 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239803+0000", 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000152962 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.564 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36776 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.327138211000005, 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069346781999999996, 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.565 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36776 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.170741+0000", 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175518+0000", 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047761949999999996 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175526+0000", 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0590000000000003e-06 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175540+0000", 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4677e-05 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175550+0000", 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6609999999999995e-06 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239834+0000", 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.064284145000000001 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.239896+0000", 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2303000000000003e-05 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.240088+0000", 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019174200000000001 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36787 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 974848~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.321991466, 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066887682000000004, 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36787 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175888+0000", 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175889+0000", 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5000000000000002e-07 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175894+0000", 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7700000000000001e-06 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175911+0000", 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7921e-05 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175920+0000", 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4209999999999995e-06 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242585+0000", 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066665355999999995 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242631+0000", 2024-09-04T21:55:58.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5779000000000003e-05 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242776+0000", 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000144785 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36788 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 991232~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:58.574 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.321948207000005, 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067126168, 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36788 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:58.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175931+0000", 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175932+0000", 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175935+0000", 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.084e-06 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175948+0000", 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2592000000000001e-05 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175954+0000", 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8260000000000004e-06 2024-09-04T21:55:58.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242805+0000", 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.066850779999999999 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.242867+0000", 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1792999999999996e-05 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243058+0000", 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000190502 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36789 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1007616~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.321904336000003, 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067310352000000004, 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.579 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36789 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175975+0000", 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175976+0000", 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0999999999999998e-07 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175979+0000", 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.072e-06 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.175995+0000", 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5922000000000001e-05 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176001+0000", 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2450000000000003e-06 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243085+0000", 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067083775999999998 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243137+0000", 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2287000000000003e-05 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243286+0000", 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014844000000000001 2024-09-04T21:55:58.583 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36790 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.321868058000007, 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067567505999999999, 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.584 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36790 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176011+0000", 2024-09-04T21:55:58.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176012+0000", 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4799999999999998e-07 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176015+0000", 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0419999999999998e-06 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176023+0000", 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7139999999999993e-06 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176029+0000", 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9660000000000001e-06 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243316+0000", 2024-09-04T21:55:58.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067287198000000006 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243376+0000", 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9936999999999999e-05 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243579+0000", 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020310099999999999 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.588 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:36791 3.11 3:8cd52af3:::rbd_data.1188e6876e06.0000000000000002:f [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 83.321841952, 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067761180000000004, 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.589 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 36791 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176038+0000", 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2e-07 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176047+0000", 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5699999999999993e-06 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176054+0000", 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2819999999999999e-06 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.176060+0000", 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8880000000000002e-06 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243605+0000", 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.067545171000000001 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243648+0000", 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2503000000000003e-05 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:54:35.243799+0000", 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015124600000000001 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.593 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:143697 3.13 3:ceb3a275:::rbd_object_map.1188388e16dc:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.127116513000001, 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.004254991, 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:58.594 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 143697 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370763+0000", 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:58.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370766+0000", 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7429999999999998e-06 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370776+0000", 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8549999999999997e-06 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:58.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370795+0000", 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9599e-05 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370807+0000", 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1652e-05 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370896+0000", 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9104000000000006e-05 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370911+0000", 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4986e-05 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370490+0000", 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99957913600000003 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.370701+0000", 2024-09-04T21:55:58.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021104399999999999 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4", 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.371058+0000", 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00035635700000000002 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374510+0000", 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034528860000000001 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374898+0000", 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038715800000000002 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.374971+0000", 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3881999999999995e-05 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:41.375018+0000", 2024-09-04T21:55:58.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6588999999999997e-05 2024-09-04T21:55:58.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.601 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:58.601 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:58.601 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:58.603 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:55:58.603 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:55:58.604 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:55:58.598+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:55:58.703 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:58.695+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:58.789 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:58.789 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:58.789 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:55:58.789 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:58.803 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:55:58.804 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:58.798+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:58.905 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:58.899+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:58.960 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:58.961 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:55:58.961 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:55:58.961 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:55:58.961 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:58.974 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:55:59.006 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:59.002+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:59.107 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:59.102+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:59.144 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23, version: 95'461, data_included: [0~262144,327680~65536,507904~15827,3260416~65536,3407872~32768,3555328~65536,3670016~32768,3850240~32768], data_size: 572883, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23@95'461, size: 3883008, copy_subset: [(0, 3883008)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3883008, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.32972169, 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060935900000000003, 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811327+0000", 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3629999999999999e-06 2024-09-04T21:55:59.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811906+0000", 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057844000000000005 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811940+0000", 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4096e-05 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811967+0000", 2024-09-04T21:55:59.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7202000000000001e-05 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812156+0000", 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189073 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812200+0000", 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3748000000000002e-05 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.817419+0000", 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052196680000000002 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2959 3.f e108/105 3:f7c4ed88:::rbd_data.1188388e16dc.0000000000000001:1c v 108'438, mlcod=107'435)", 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 18.776399671, 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055309019999999999, 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.149 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364649+0000", 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364676+0000", 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7135e-05 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364688+0000", 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2082000000000001e-05 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364700+0000", 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1885999999999999e-05 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364799+0000", 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9593999999999995e-05 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364831+0000", 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1628000000000003e-05 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370108+0000", 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052768629999999997 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370179+0000", 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1046999999999999e-05 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.153 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3025 3.f e113/105 3:f69cda90:::rbd_data.1188e5a3a295.0000000000000004:18 v 113'446, mlcod=111'443)", 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.736859435, 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070421360000000001, 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404189+0000", 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404195+0000", 2024-09-04T21:55:59.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3149999999999997e-06 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404204+0000", 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0240000000000003e-06 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404213+0000", 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5040000000000008e-06 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404315+0000", 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010159599999999999 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404344+0000", 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9175000000000001e-05 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411189+0000", 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006844854 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411230+0000", 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1137999999999998e-05 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.3.0:4041 3.19 e113/105 3:98fe2de1:::rbd_data.1188e5a3a295.000000000000002a:18 v 113'1164, mlcod=110'1161)", 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.736788459, 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007607361, 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.158 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:55:59.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404260+0000", 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404265+0000", 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0119999999999996e-06 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404274+0000", 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.764e-06 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404282+0000", 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1169999999999998e-06 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404538+0000", 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025666600000000001 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404564+0000", 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5415999999999999e-05 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411833+0000", 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0072691520000000001 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411866+0000", 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3643000000000003e-05 2024-09-04T21:55:59.162 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3026 3.f e113/105 3:f7074c4a:::rbd_data.1188e5a3a295.0000000000000024:18 v 113'448, mlcod=111'443)", 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.736392983, 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073657970000000003, 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4799999999999997e-07 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404663+0000", 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404675+0000", 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2108e-05 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404683+0000", 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2719999999999997e-06 2024-09-04T21:55:59.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405007+0000", 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032386699999999998 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405035+0000", 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7795000000000001e-05 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411899+0000", 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068636569999999996 2024-09-04T21:55:59.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.412020+0000", 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121777 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144606 3.14 3:2afd88fa:::rbd_data.1188c8e6983d.0000000000000000:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.602358215000001, 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010126151, 2024-09-04T21:55:59.167 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144606 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538692+0000", 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2340000000000001e-06 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538701+0000", 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7829999999999993e-06 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538755+0000", 2024-09-04T21:55:59.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3352e-05 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538770+0000", 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5027e-05 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538877+0000", 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010716 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539557+0000", 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067977599999999999 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539848+0000", 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000291253 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544640+0000", 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047919900000000003 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548544+0000", 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039044190000000001 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548701+0000", 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015645599999999999 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548815+0000", 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114701 2024-09-04T21:55:59.173 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144607 3.1c 3:3e16db79:::rbd_data.1188c8e6983d.0000000000000001:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.602247647, 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070450139999999996, 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.174 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144607 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538801+0000", 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0070000000000001e-06 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538806+0000", 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.746e-06 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538819+0000", 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2894000000000001e-05 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538827+0000", 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1820000000000006e-06 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538963+0000", 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135981 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539501+0000", 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053799499999999999 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539807+0000", 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030629000000000002 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544657+0000", 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048502149999999997 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545514+0000", 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085667800000000004 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545676+0000", 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016225300000000001 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545845+0000", 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016877299999999999 2024-09-04T21:55:59.179 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144613 3.1c 3:38cae06c:::rbd_data.1188c8e6983d.0000000000000013:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.602158422, 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0099469250000000006, 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.180 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144613 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:55:59.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538898+0000", 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538906+0000", 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3659999999999992e-06 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538930+0000", 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4079000000000001e-05 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538952+0000", 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1437000000000002e-05 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542268+0000", 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033164679999999999 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542700+0000", 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043201699999999998 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543018+0000", 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031757600000000002 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545938+0000", 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002920387 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548603+0000", 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026646299999999999 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548736+0000", 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013379 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548836+0000", 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9535000000000002e-05 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:144608 3.b e114/105 3:d125cd18:::rbd_data.1188c8e6983d.0000000000000002:head v 114'334, mlcod=106'332)", 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.598852768, 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005416677, 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.186 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542197+0000", 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3870000000000002e-06 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542213+0000", 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6362999999999999e-05 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542243+0000", 2024-09-04T21:55:59.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9833e-05 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542260+0000", 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6903000000000001e-05 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543925+0000", 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001664741 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543988+0000", 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3335000000000003e-05 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547491+0000", 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035024880000000002 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547611+0000", 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120627 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.190 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144621 3.1c 3:3a3265fe:::rbd_data.1188c8e6983d.000000000000002e:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.594895376, 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004689365, 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.191 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144621 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546152+0000", 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546156+0000", 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.523e-06 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546172+0000", 2024-09-04T21:55:59.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6353000000000001e-05 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546208+0000", 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6269000000000001e-05 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546230+0000", 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1849999999999999e-05 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546385+0000", 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015487699999999999 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.546843+0000", 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045768300000000002 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547462+0000", 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061928999999999999 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.549650+0000", 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021879289999999999 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550617+0000", 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096704600000000005 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550743+0000", 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012623300000000001 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.550841+0000", 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8312000000000003e-05 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:145393 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_remove_image in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:54.499302+0000", 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6417453569999996, 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070046400000000003, 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.197 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 145393 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499302+0000", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499302+0000", 2024-09-04T21:55:59.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499304+0000", 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6559999999999999e-06 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499312+0000", 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8609999999999998e-06 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499329+0000", 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7067000000000001e-05 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499342+0000", 2024-09-04T21:55:59.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3397999999999999e-05 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499406+0000", 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4183000000000005e-05 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499423+0000", 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6739999999999999e-05 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.502191+0000", 2024-09-04T21:55:59.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002768108 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.502318+0000", 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127021 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:59.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.502572+0000", 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000253918 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.503331+0000", 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075931600000000001 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.506200+0000", 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028681510000000002 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.506248+0000", 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8418000000000003e-05 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.506307+0000", 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8802999999999999e-05 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:145522 3.1d 3:b91e1d12:::rbd_header.11882296a3bb:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:55.366616+0000", 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.774431678, 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.017559646000000002, 2024-09-04T21:55:59.206 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 145522 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366616+0000", 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366616+0000", 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366620+0000", 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7929999999999999e-06 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366637+0000", 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:55:59.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366676+0000", 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9242999999999997e-05 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366701+0000", 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4876e-05 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366871+0000", 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017062100000000001 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.368549+0000", 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016772530000000001 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.372950+0000", 2024-09-04T21:55:59.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044011659999999998 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.382081+0000", 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0091311719999999999 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.383901+0000", 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001819874 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.384059+0000", 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015797700000000001 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.384175+0000", 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000116545 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3131 3.13 e123/108 3:c99e10a9:::rbd_data.11882296a3bb.00000000000001d9:12 v 123'952, mlcod=117'947)", 2024-09-04T21:55:59.212 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:55.386264+0000", 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.7547832620000001, 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074059190000000004, 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386264+0000", 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386264+0000", 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386266+0000", 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8530000000000001e-06 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386279+0000", 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2957e-05 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386303+0000", 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4034999999999999e-05 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386320+0000", 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7244e-05 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.387208+0000", 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088717699999999995 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.387260+0000", 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2531999999999998e-05 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.393608+0000", 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0063484420000000001 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.393670+0000", 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1679000000000004e-05 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.216 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:145624 3.1b 3:dbc23d5c:::rbd_data.11882296a3bb.000000000000003d:head [delete] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:55.558770+0000", 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.5822775849999999, 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061668080000000002, 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.217 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 145624 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558770+0000", 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558770+0000", 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558773+0000", 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558786+0000", 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3353000000000001e-05 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558818+0000", 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1860000000000003e-05 2024-09-04T21:55:59.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558839+0000", 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0706000000000001e-05 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.559033+0000", 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019353 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.559628+0000", 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00059584600000000003 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.559829+0000", 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020036300000000001 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.561505+0000", 2024-09-04T21:55:59.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001676117 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.564661+0000", 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031556879999999998 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.564825+0000", 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016441399999999999 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.564937+0000", 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011178799999999999 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146116 3.3 3:c7899f10:::rbd_data.1188e6876e06.0000000000000001:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646128+0000", 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49491901700000002, 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089224379999999995, 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146116 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646128+0000", 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646128+0000", 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646131+0000", 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9699999999999999e-06 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646145+0000", 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.38e-05 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646177+0000", 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2100000000000001e-05 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646201+0000", 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3502999999999998e-05 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646378+0000", 2024-09-04T21:55:59.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017719 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646710+0000", 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033145199999999998 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6", 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646895+0000", 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018562299999999999 2024-09-04T21:55:59.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654829+0000", 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0079342010000000001 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654926+0000", 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6367999999999996e-05 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654992+0000", 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6472999999999999e-05 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655051+0000", 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8758000000000002e-05 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146119 3.1 3:82d76238:::rbd_data.1188e6876e06.0000000000000004:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646254+0000", 2024-09-04T21:55:59.229 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49479351500000002, 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0081848040000000004, 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146119 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646254+0000", 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646254+0000", 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646256+0000", 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.59e-06 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646261+0000", 2024-09-04T21:55:59.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3859999999999998e-06 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646290+0000", 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.889e-05 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646309+0000", 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9313e-05 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646616+0000", 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030682099999999998 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647075+0000", 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045925600000000001 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:59.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647371+0000", 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000296252 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.651556+0000", 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041840999999999996 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654177+0000", 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026217430000000002 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654331+0000", 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000153731 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654439+0000", 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107722 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146122 3.3 3:c1d3c829:::rbd_data.1188e6876e06.0000000000000007:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:55:59.235 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646353+0000", 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49469479700000002, 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0094368070000000002, 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146122 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646353+0000", 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646353+0000", 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646354+0000", 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.497e-06 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646361+0000", 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6000000000000003e-06 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646381+0000", 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9986000000000001e-05 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646397+0000", 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6087e-05 2024-09-04T21:55:59.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.651606+0000", 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005208776 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.652113+0000", 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000507195 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6", 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.652515+0000", 2024-09-04T21:55:59.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040166000000000003 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654894+0000", 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002379286 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655539+0000", 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064523100000000002 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655681+0000", 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014168699999999999 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655790+0000", 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010880200000000001 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146123 3.1 3:82fd2c63:::rbd_data.1188e6876e06.0000000000000008:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646431+0000", 2024-09-04T21:55:59.241 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49461650499999998, 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011851346, 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146123 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646431+0000", 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646431+0000", 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646432+0000", 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4100000000000001e-06 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646438+0000", 2024-09-04T21:55:59.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.84e-06 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646458+0000", 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0207000000000002e-05 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646474+0000", 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5716e-05 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.648691+0000", 2024-09-04T21:55:59.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002216946 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.650120+0000", 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014286870000000001 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.650461+0000", 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034112999999999998 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654886+0000", 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044252370000000003 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.657993+0000", 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003107098 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.658162+0000", 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169214 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.658282+0000", 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119861 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:146115 3.11 e126/105 3:8c17d657:::rbd_data.1188e6876e06.0000000000000000:head v 126'541, mlcod=125'539)", 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.647100+0000", 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49394732899999999, 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0077587740000000004, 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.247 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647100+0000", 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647100+0000", 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647102+0000", 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2400000000000002e-06 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647117+0000", 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4876e-05 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647309+0000", 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000192201 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647328+0000", 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8615999999999999e-05 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647540+0000", 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021141000000000001 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647596+0000", 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6535e-05 2024-09-04T21:55:59.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654743+0000", 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0071471679999999998 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654859+0000", 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115728 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.251 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146130 3.1d 3:ba4c3727:::rbd_data.1188e6876e06.0000000000000011:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.655516+0000", 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.48553108900000003, 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061256540000000003, 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146130 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655516+0000", 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655516+0000", 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655520+0000", 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655534+0000", 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3933e-05 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655565+0000", 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1155999999999998e-05 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655587+0000", 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1630000000000001e-05 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655747+0000", 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016034 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.656193+0000", 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044614600000000002 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:55:59.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.656570+0000", 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037673699999999998 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.660364+0000", 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037937600000000002 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.661299+0000", 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00093472399999999995 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.661520+0000", 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022108300000000001 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.661642+0000", 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012240499999999999 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:55:59.257 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:55:59.259 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:55:59.251+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:55:59.259 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:55:59.360 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:55:59.354+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:55:59.391 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:59.392 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:59.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:55:59.461 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:55:59.455+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:55:59.562 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:59.555+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:59.582 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:55:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:55:59.583 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:59.598 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:55:59.636 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-04T21:55:59.637 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:55:59.638 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-04T21:55:59.638 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-04T21:55:59.695 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:55:59.690+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:55:59.728 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:55:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:55:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:55:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050222694, 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:55:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:55:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050195909, 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050166976, 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:55:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:55:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:55:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.05014332899999, 2024-09-04T21:55:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:55:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:55:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050123548, 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.05010415300001, 2024-09-04T21:55:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:55:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:55:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:55:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.05008488, 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.05006565799999, 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:55:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:55:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050046821, 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:55:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:55:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050027418, 2024-09-04T21:55:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.050006362, 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:55:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:55:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.04997418400001, 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:55:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.04994796299999, 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:55:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.049919754, 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:55:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.04989971400001, 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:55:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.049880332, 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:55:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.049856744, 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:55:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.049836427, 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:55:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:55:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.049816875, 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:55:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:55:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.04979711599999, 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:55:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:55:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:55:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:55:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:55:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:55:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:55:59.831 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:55:59.832 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:55:59.833 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:55:59.826+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:55:59.933 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:55:59.931+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:55:59.973 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-04T21:55:59.966+0000 7f1d9e303640 -1 mon.a@0(leader).osd e127 definitely_dead 0 2024-09-04T21:55:59.994 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:55:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:55:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:55:59.994 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:00.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:56:00.058 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:00.054+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:00.126 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:00.126 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:00.126 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:00.126 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:00.126 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:00.135 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:00.127+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:00.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:56:00.236 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:00.231+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.453484175, 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:56:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:56:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:56:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.327436912, 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:56:00.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:56:00.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:56:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:56:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.080229305, 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:56:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:56:00.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:56:00.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.05247151499999, 2024-09-04T21:56:00.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.052355953, 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.336 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:56:00.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.593048616, 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.342 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:00.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:56:00.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.59282255799999, 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:56:00.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:56:00.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7387 3.4 3:22d619b6:::rbd_data.1188b1294ada.000000000000004b:head [set-alloc-hint object_size 2097152 write_size 2097152,write 547721~28095] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 113.11560949, 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038326699999999998, 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.354 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7387 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189596+0000", 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189600+0000", 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2969999999999999e-06 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189633+0000", 2024-09-04T21:56:00.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3238999999999998e-05 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189667+0000", 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3766999999999997e-05 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189689+0000", 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2187e-05 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.189827+0000", 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000138095 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190053+0000", 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226462 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.190310+0000", 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025605199999999999 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.191833+0000", 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015238910000000001 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193266+0000", 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001432801 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193357+0000", 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0920000000000004e-05 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.193429+0000", 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1958999999999993e-05 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.88622137900001, 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:56:00.360 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:56:00.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:56:00.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.587917883, 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:56:00.366 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:56:00.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:56:00.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 112.40817850400001, 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:56:00.372 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:56:00.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:56:00.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 111.741851599, 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.378 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:56:00.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:56:00.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:145127 3.19 3.1e5da059 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:00.384 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.846705619, 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90329116399999998, 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 145127 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458503+0000", 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0029999999999999e-06 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458512+0000", 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3590000000000008e-06 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458530+0000", 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8790000000000001e-05 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458539+0000", 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2290000000000003e-06 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458630+0000", 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0841000000000007e-05 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458644+0000", 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361754+0000", 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90310956399999998 2024-09-04T21:56:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361791+0000", 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7728999999999997e-05 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4160 3.8 e122/121 3:1505c1f6:::rbd_data.1188b450424.0000000000000003:14 v 122'534, mlcod=0'0)", 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.8130255469999996, 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0042635119999999997, 2024-09-04T21:56:00.389 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492182+0000", 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492200+0000", 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8009e-05 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492229+0000", 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8469000000000001e-05 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492249+0000", 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9884000000000001e-05 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492350+0000", 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000101572 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492378+0000", 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8138999999999999e-05 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496377+0000", 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0039985480000000002 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496444+0000", 2024-09-04T21:56:00.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6828000000000005e-05 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4161 3.8 e122/121 3:13882358:::rbd_data.1188b450424.0000000000000004:14 v 122'536, mlcod=0'0)", 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.8102414939999996, 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057478820000000002, 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.394 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494967+0000", 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494982+0000", 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5226e-05 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495010+0000", 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8143000000000001e-05 2024-09-04T21:56:00.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495029+0000", 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8872000000000002e-05 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495238+0000", 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020955799999999999 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495292+0000", 2024-09-04T21:56:00.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3548999999999998e-05 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500587+0000", 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052953439999999996 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500712+0000", 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012510000000000001 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4171 3.8 e123/121 3:16896659:::rbd_data.11882296a3bb.00000000000001d8:12 v 123'538, mlcod=122'534)", 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.9421087000000004, 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053660619999999996, 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:00.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363098+0000", 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363103+0000", 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9269999999999996e-06 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363114+0000", 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1077000000000001e-05 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363137+0000", 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363420+0000", 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028287199999999998 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363445+0000", 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5471999999999999e-05 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368381+0000", 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0049356000000000001 2024-09-04T21:56:00.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368463+0000", 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2495000000000002e-05 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:276 3.1a e123/120 3:58e50b11:::rbd_data.11882296a3bb.000000000000004e:12 v 123'923, mlcod=116'919)", 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.9403226360000003, 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059945689999999999, 2024-09-04T21:56:00.403 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364884+0000", 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9599999999999999e-07 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364893+0000", 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6000000000000007e-06 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364919+0000", 2024-09-04T21:56:00.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6156000000000002e-05 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364936+0000", 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6920000000000001e-05 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365020+0000", 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4146000000000004e-05 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365036+0000", 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6277e-05 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370836+0000", 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058004290000000002 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370878+0000", 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1344999999999998e-05 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146120 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.658819021, 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040719019999999996, 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.408 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146120 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:00.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646390+0000", 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0630000000000002e-06 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646410+0000", 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9749999999999999e-05 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646453+0000", 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2879000000000001e-05 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646476+0000", 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3886000000000001e-05 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646633+0000", 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015649199999999999 2024-09-04T21:56:00.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646831+0000", 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019817899999999999 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646996+0000", 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000164745 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.648404+0000", 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001408149 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650116+0000", 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001711775 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650329+0000", 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021338100000000001 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650459+0000", 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129603 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146127 3.0 3:04d53328:::rbd_data.1188e6876e06.000000000000000e:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.649956663, 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038955439999999999, 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.414 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146127 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655252+0000", 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7269999999999999e-06 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655263+0000", 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0893000000000001e-05 2024-09-04T21:56:00.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655294+0000", 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1479000000000001e-05 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655310+0000", 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5625999999999999e-05 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655473+0000", 2024-09-04T21:56:00.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016301900000000001 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655790+0000", 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000317492 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.656104+0000", 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031342799999999998 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.657718+0000", 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001614571 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.658939+0000", 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012200869999999999 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.659027+0000", 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8450999999999997e-05 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.659145+0000", 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011777099999999999 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146276 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:head [delete] snapc e=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.61162141199999998, 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051386740000000002, 2024-09-04T21:56:00.420 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146276 2024-09-04T21:56:00.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:00.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693587+0000", 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5600000000000001e-06 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693595+0000", 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2640000000000008e-06 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693618+0000", 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3051999999999999e-05 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693630+0000", 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2074e-05 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693705+0000", 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4867e-05 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693851+0000", 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014598700000000001 2024-09-04T21:56:00.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693977+0000", 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012523100000000001 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.695244+0000", 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001267518 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698604+0000", 2024-09-04T21:56:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003359608 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698671+0000", 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7810000000000003e-05 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698723+0000", 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1703000000000003e-05 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:00.427 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:00.427 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:00.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:56:00.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:56:00.430 INFO:teuthology.orchestra.run.smithi112.stderr:marked in osd.0. 2024-09-04T21:56:00.430 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-04T21:56:00.430 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:00.422+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:00.529 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:00.522+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:00.593 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:00.593 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:00.593 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:00.593 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:00.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:56:00.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:56:00.614 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:00.615 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:00.630 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:00.623+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:00.632 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:00.632 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:00.649 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:00.649 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:00.668 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:00.668 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:00.689 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:00.689 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:00.710 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:00.710 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:00.730 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:00.731 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:00.731 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:00.723+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:00.736 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:00.736 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:00.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:56:00.757 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:00.758 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:00.831 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:00.826+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:00.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:56:00.962 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:00.958+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:01.029 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.029 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.033 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:01.026+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:01.044 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:56:01.134 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:01.130+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:01.202 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.202 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.214 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:56:01.258 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:01.254+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:01.335 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:01.331+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:01.366 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:56:01.369 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.369 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.394 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:56:01.442 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:01.438+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:01.524 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.524 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.537 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:01.531+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:01.538 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:56:01.639 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:01.634+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:01.703 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.703 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.717 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:56:01.739 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:01.738+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:01.840 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.840 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.840 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:01.834+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:01.854 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:56:01.941 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:01.935+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:01.977 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:01.978 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:01.992 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:56:02.041 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:02.038+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:02.142 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:02.135+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:02.161 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:02.161 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:02.175 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:56:02.263 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:02.259+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:02.344 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:02.344 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:02.344 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:02.338+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:02.358 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:56:02.447 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:02.443+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:02.528 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:02.529 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:02.543 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:56:02.588 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:02.583+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:02.654 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:02.642+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:02.715 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:02.715 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:02.731 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:56:02.775 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:02.771+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:02.783 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:56:02.848 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:02.842+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:02.860 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:02.861 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:02.876 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:56:02.949 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:02.942+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:02.961 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:02.961 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:02.982 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:02.982 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:02.999 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:03.018 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:03.019 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:03.038 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:03.038 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:03.039 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:56:03.039 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:56:03.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPushReply(3.1b 130/129 [PushReplyOp(3:da4215d1:::rbd_data.118892c64562.0000000000000001:c)])", 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:56:03.038330+0000", 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.00086481899999999996, 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00090933500000000005, 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038330+0000", 2024-09-04T21:56:03.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038330+0000", 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038331+0000", 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.04e-06 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038337+0000", 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6099999999999997e-06 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038351+0000", 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4344e-05 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038359+0000", 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3240000000000003e-06 2024-09-04T21:56:03.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:03.038428+0000", 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8032000000000002e-05 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 1 2024-09-04T21:56:03.043 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:56:03.050 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:03.046+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:03.055 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:56:03.059 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:03.060 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:03.080 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:03.081 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:03.100 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:03.100 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:03.166 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:03.162+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:03.186 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:56:03.186 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:56:03.186 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:56:03.186 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:56:03.186 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:56:03.200 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:56:03.252 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:03.247+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:03.322 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:56:03.322 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-04T21:56:03.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-04T21:56:03.322 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-04T21:56:03.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.1d 106/105 [PushOp(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23, version: 95'461, data_included: [0~262144,327680~65536,507904~15827,3260416~65536,3407872~32768,3555328~65536,3670016~32768,3850240~32768], data_size: 572883, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:ba759573:::rbd_data.1188f71f571e.0000000000000006:23@95'461, size: 3883008, copy_subset: [(0, 3883008)], clone_subset: {}, snapset: 23=[]:{23: [23]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3883008, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 24.507474247000001, 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0060935900000000003, 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811326+0000", 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811327+0000", 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3629999999999999e-06 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811906+0000", 2024-09-04T21:56:03.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057844000000000005 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811940+0000", 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4096e-05 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.811967+0000", 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7202000000000001e-05 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812156+0000", 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000189073 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.812200+0000", 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3748000000000002e-05 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:38.817419+0000", 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052196680000000002 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:2959 3.f e108/105 3:f7c4ed88:::rbd_data.1188388e16dc.0000000000000001:1c v 108'438, mlcod=107'435)", 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.954152228000002, 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0055309019999999999, 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:56:03.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364648+0000", 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364649+0000", 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6700000000000003e-07 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364676+0000", 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7135e-05 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364688+0000", 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2082000000000001e-05 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364700+0000", 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1885999999999999e-05 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364799+0000", 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9593999999999995e-05 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.364831+0000", 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1628000000000003e-05 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370108+0000", 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0052768629999999997 2024-09-04T21:56:03.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:40.370179+0000", 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1046999999999999e-05 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3025 3.f e113/105 3:f69cda90:::rbd_data.1188e5a3a295.0000000000000004:18 v 113'446, mlcod=111'443)", 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 17.914611992000001, 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070421360000000001, 2024-09-04T21:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404188+0000", 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404189+0000", 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404195+0000", 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3149999999999997e-06 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404204+0000", 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0240000000000003e-06 2024-09-04T21:56:03.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404213+0000", 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5040000000000008e-06 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404315+0000", 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010159599999999999 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404344+0000", 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9175000000000001e-05 2024-09-04T21:56:03.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411189+0000", 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.006844854 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411230+0000", 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1137999999999998e-05 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.3.0:4041 3.19 e113/105 3:98fe2de1:::rbd_data.1188e5a3a295.000000000000002a:18 v 113'1164, mlcod=110'1161)", 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 17.914541016000001, 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.007607361, 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404259+0000", 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404260+0000", 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9100000000000004e-07 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404265+0000", 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0119999999999996e-06 2024-09-04T21:56:03.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404274+0000", 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.764e-06 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404282+0000", 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1169999999999998e-06 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404538+0000", 2024-09-04T21:56:03.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025666600000000001 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404564+0000", 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5415999999999999e-05 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411833+0000", 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0072691520000000001 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411866+0000", 2024-09-04T21:56:03.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3643000000000003e-05 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3026 3.f e113/105 3:f7074c4a:::rbd_data.1188e5a3a295.0000000000000024:18 v 113'448, mlcod=111'443)", 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 17.91414554, 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0073657970000000003, 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.340 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404655+0000", 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4799999999999997e-07 2024-09-04T21:56:03.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404663+0000", 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404675+0000", 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2108e-05 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.404683+0000", 2024-09-04T21:56:03.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2719999999999997e-06 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405007+0000", 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032386699999999998 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.405035+0000", 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7795000000000001e-05 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.411899+0000", 2024-09-04T21:56:03.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0068636569999999996 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:45.412020+0000", 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121777 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144606 3.14 3:2afd88fa:::rbd_data.1188c8e6983d.0000000000000000:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.780110772, 2024-09-04T21:56:03.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010126151, 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144606 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538689+0000", 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538692+0000", 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2340000000000001e-06 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538701+0000", 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7829999999999993e-06 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538755+0000", 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3352e-05 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538770+0000", 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5027e-05 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538877+0000", 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010716 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539557+0000", 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067977599999999999 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539848+0000", 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000291253 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544640+0000", 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0047919900000000003 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548544+0000", 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0039044190000000001 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548701+0000", 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015645599999999999 2024-09-04T21:56:03.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548815+0000", 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114701 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144607 3.1c 3:3e16db79:::rbd_data.1188c8e6983d.0000000000000001:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.780000204, 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070450139999999996, 2024-09-04T21:56:03.350 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144607 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538800+0000", 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538801+0000", 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0070000000000001e-06 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538806+0000", 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.746e-06 2024-09-04T21:56:03.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538819+0000", 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2894000000000001e-05 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538827+0000", 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1820000000000006e-06 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538963+0000", 2024-09-04T21:56:03.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000135981 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539501+0000", 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053799499999999999 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.539807+0000", 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030629000000000002 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.544657+0000", 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048502149999999997 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545514+0000", 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00085667800000000004 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545676+0000", 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016225300000000001 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545845+0000", 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016877299999999999 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:144613 3.1c 3:38cae06c:::rbd_data.1188c8e6983d.0000000000000013:head [delete] snapc 17=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.779910979, 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0099469250000000006, 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.356 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 144613 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:56:03.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538889+0000", 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538898+0000", 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538906+0000", 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3659999999999992e-06 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538930+0000", 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4079000000000001e-05 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.538952+0000", 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1437000000000002e-05 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542268+0000", 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0033164679999999999 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542700+0000", 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043201699999999998 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543018+0000", 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031757600000000002 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.545938+0000", 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002920387 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548603+0000", 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026646299999999999 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548736+0000", 2024-09-04T21:56:03.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013379 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.548836+0000", 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9535000000000002e-05 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:144608 3.b e114/105 3:d125cd18:::rbd_data.1188c8e6983d.0000000000000002:head v 114'334, mlcod=106'332)", 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.776605324999998, 2024-09-04T21:56:03.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005416677, 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542195+0000", 2024-09-04T21:56:03.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542197+0000", 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3870000000000002e-06 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542213+0000", 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6362999999999999e-05 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542243+0000", 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9833e-05 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.542260+0000", 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6903000000000001e-05 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543925+0000", 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001664741 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.543988+0000", 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3335000000000003e-05 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547491+0000", 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035024880000000002 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:46.547611+0000", 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120627 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:145393 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_remove_image in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:54.499302+0000", 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8194979139999994, 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0070046400000000003, 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.367 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 145393 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499302+0000", 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499302+0000", 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499304+0000", 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6559999999999999e-06 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499312+0000", 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8609999999999998e-06 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499329+0000", 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7067000000000001e-05 2024-09-04T21:56:03.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499342+0000", 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3397999999999999e-05 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499406+0000", 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4183000000000005e-05 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.499423+0000", 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6739999999999999e-05 2024-09-04T21:56:03.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.502191+0000", 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002768108 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.502318+0000", 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127021 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.502572+0000", 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000253918 2024-09-04T21:56:03.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.503331+0000", 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075931600000000001 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.506200+0000", 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0028681510000000002 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.506248+0000", 2024-09-04T21:56:03.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8418000000000003e-05 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:54.506307+0000", 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8802999999999999e-05 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:145522 3.1d 3:b91e1d12:::rbd_header.11882296a3bb:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:55.366616+0000", 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.9521842349999998, 2024-09-04T21:56:03.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.017559646000000002, 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 145522 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366616+0000", 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366616+0000", 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366620+0000", 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7929999999999999e-06 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366637+0000", 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7125999999999998e-05 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366676+0000", 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9242999999999997e-05 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366701+0000", 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4876e-05 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.366871+0000", 2024-09-04T21:56:03.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017062100000000001 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.368549+0000", 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016772530000000001 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.372950+0000", 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044011659999999998 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.382081+0000", 2024-09-04T21:56:03.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0091311719999999999 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.383901+0000", 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001819874 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.384059+0000", 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015797700000000001 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.384175+0000", 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000116545 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3131 3.13 e123/108 3:c99e10a9:::rbd_data.11882296a3bb.00000000000001d9:12 v 123'952, mlcod=117'947)", 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:55.386264+0000", 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.9325358189999999, 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0074059190000000004, 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.379 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386264+0000", 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386264+0000", 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386266+0000", 2024-09-04T21:56:03.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8530000000000001e-06 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386279+0000", 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2957e-05 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386303+0000", 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4034999999999999e-05 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.386320+0000", 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7244e-05 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.387208+0000", 2024-09-04T21:56:03.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088717699999999995 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.387260+0000", 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2531999999999998e-05 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.393608+0000", 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0063484420000000001 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.393670+0000", 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1679000000000004e-05 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:145624 3.1b 3:dbc23d5c:::rbd_data.11882296a3bb.000000000000003d:head [delete] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:55.558770+0000", 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 7.7600301419999997, 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061668080000000002, 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.387 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 145624 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558770+0000", 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558770+0000", 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558773+0000", 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558786+0000", 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3353000000000001e-05 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558818+0000", 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1860000000000003e-05 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.558839+0000", 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0706000000000001e-05 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.559033+0000", 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019353 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.559628+0000", 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00059584600000000003 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7", 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.559829+0000", 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020036300000000001 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.561505+0000", 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001676117 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.564661+0000", 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031556879999999998 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.564825+0000", 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016441399999999999 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:55.564937+0000", 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011178799999999999 2024-09-04T21:56:03.392 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146116 3.3 3:c7899f10:::rbd_data.1188e6876e06.0000000000000001:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646128+0000", 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6726715739999998, 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0089224379999999995, 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.393 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146116 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646128+0000", 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646128+0000", 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646131+0000", 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9699999999999999e-06 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646145+0000", 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.38e-05 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646177+0000", 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2100000000000001e-05 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646201+0000", 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3502999999999998e-05 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646378+0000", 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017719 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646710+0000", 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033145199999999998 2024-09-04T21:56:03.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6", 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646895+0000", 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018562299999999999 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654829+0000", 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0079342010000000001 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654926+0000", 2024-09-04T21:56:03.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6367999999999996e-05 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654992+0000", 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6472999999999999e-05 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655051+0000", 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8758000000000002e-05 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146119 3.1 3:82d76238:::rbd_data.1188e6876e06.0000000000000004:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646254+0000", 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6725460720000003, 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0081848040000000004, 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146119 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.399 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646254+0000", 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646254+0000", 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646256+0000", 2024-09-04T21:56:03.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.59e-06 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646261+0000", 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3859999999999998e-06 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646290+0000", 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.889e-05 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646309+0000", 2024-09-04T21:56:03.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9313e-05 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646616+0000", 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030682099999999998 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647075+0000", 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045925600000000001 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:03.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647371+0000", 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000296252 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.651556+0000", 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0041840999999999996 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654177+0000", 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026217430000000002 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654331+0000", 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000153731 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654439+0000", 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107722 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146122 3.3 3:c1d3c829:::rbd_data.1188e6876e06.0000000000000007:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646353+0000", 2024-09-04T21:56:03.404 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.672447354, 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0094368070000000002, 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146122 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646353+0000", 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646353+0000", 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646354+0000", 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.497e-06 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646361+0000", 2024-09-04T21:56:03.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6000000000000003e-06 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646381+0000", 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9986000000000001e-05 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646397+0000", 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6087e-05 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.651606+0000", 2024-09-04T21:56:03.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.005208776 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.652113+0000", 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000507195 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6", 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.652515+0000", 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040166000000000003 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654894+0000", 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002379286 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655539+0000", 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064523100000000002 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655681+0000", 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014168699999999999 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655790+0000", 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010880200000000001 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146123 3.1 3:82fd2c63:::rbd_data.1188e6876e06.0000000000000008:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.646431+0000", 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6723690619999996, 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011851346, 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.410 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146123 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646431+0000", 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646431+0000", 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646432+0000", 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4100000000000001e-06 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646438+0000", 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.84e-06 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646458+0000", 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0207000000000002e-05 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.646474+0000", 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5716e-05 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.648691+0000", 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002216946 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.650120+0000", 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014286870000000001 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.650461+0000", 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034112999999999998 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654886+0000", 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0044252370000000003 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.657993+0000", 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003107098 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.658162+0000", 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000169214 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.658282+0000", 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119861 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.415 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4488.0:146115 3.11 e126/105 3:8c17d657:::rbd_data.1188e6876e06.0000000000000000:head v 126'541, mlcod=125'539)", 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.647100+0000", 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6716998859999999, 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0077587740000000004, 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647100+0000", 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647100+0000", 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647102+0000", 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2400000000000002e-06 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647117+0000", 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4876e-05 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647309+0000", 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000192201 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647328+0000", 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8615999999999999e-05 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647540+0000", 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00021141000000000001 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.647596+0000", 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6535e-05 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654743+0000", 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0071471679999999998 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.654859+0000", 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115728 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4488.0:146130 3.1d 3:ba4c3727:::rbd_data.1188e6876e06.0000000000000011:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:55:58.655516+0000", 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.663283646, 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0061256540000000003, 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.420 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4488", 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 146130 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655516+0000", 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655516+0000", 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655520+0000", 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655534+0000", 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3933e-05 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655565+0000", 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1155999999999998e-05 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655587+0000", 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1630000000000001e-05 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.655747+0000", 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016034 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.656193+0000", 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044614600000000002 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5", 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.656570+0000", 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037673699999999998 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.660364+0000", 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037937600000000002 2024-09-04T21:56:03.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.661299+0000", 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00093472399999999995 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.661520+0000", 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022108300000000001 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:55:58.661642+0000", 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012240499999999999 2024-09-04T21:56:03.425 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.7.0:3204 3.e e128/108 3:70472631:::rbd_data.1188c517ef04.0000000000000011:d v 128'383, mlcod=122'380)", 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-04T21:56:00.373937+0000", 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 2.9448634670000002, 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.011041700999999999, 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.373937+0000", 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.373937+0000", 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.373938+0000", 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.767e-06 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.374022+0000", 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3719000000000003e-05 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.374045+0000", 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2489000000000001e-05 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.374067+0000", 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2333000000000001e-05 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.374355+0000", 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028761200000000002 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.374415+0000", 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0035999999999999e-05 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.384722+0000", 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010307652 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-04T21:56:03.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-04T21:56:00.384978+0000", 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025609299999999999 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-04T21:56:03.430 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:56:03.431 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:56:03.432 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:03.427+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:03.432 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:56:03.533 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:03.527+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146632 3.6 3.b74753e6 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.124383676, 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.124404765, 2024-09-04T21:56:03.572 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "waiting for peered", 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146632 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447333+0000", 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.66e-06 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447339+0000", 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0480000000000004e-06 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447354+0000", 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4945e-05 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447363+0000", 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0329999999999997e-06 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447449+0000", 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6327000000000002e-05 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447460+0000", 2024-09-04T21:56:03.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1173e-05 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 1 2024-09-04T21:56:03.577 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:03.591 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:56:03.638 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:03.634+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:03.723 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:03.723 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:03.724 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:03.724 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:03.735 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:03.730+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:03.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:56:03.836 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:03.831+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:03.903 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:03.903 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:03.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6452 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 444416~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22535861, 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039307531, 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.904 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6452 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675519+0000", 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2899999999999999e-07 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675527+0000", 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7160000000000003e-06 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675533+0000", 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7390000000000002e-06 2024-09-04T21:56:03.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675538+0000", 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8670000000000002e-06 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714603+0000", 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039064860999999999 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714646+0000", 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3114000000000001e-05 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714826+0000", 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017990500000000001 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:03.908 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225331825, 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:56:03.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225302892, 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:56:03.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:03.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225279245, 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:03.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225259464, 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.922 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22524006899999, 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:56:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:56:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225220796, 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:56:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225201574, 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.936 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:56:03.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225182737, 2024-09-04T21:56:03.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:56:03.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:56:03.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225163334, 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:03.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22514227800001, 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.949 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.953 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22511009999999, 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:56:03.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225083879, 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.958 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:56:03.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22505567, 2024-09-04T21:56:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.966 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22503562999999, 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.967 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:56:03.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.225016248, 2024-09-04T21:56:03.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:56:03.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.975 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22499266, 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.976 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:56:03.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:56:03.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.224972343, 2024-09-04T21:56:03.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:56:03.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:56:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.22495279100001, 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.985 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.224933032, 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:03.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:56:03.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:03.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:03.994 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:03.995 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:56:03.996 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:03.987+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:04.096 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:04.091+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:04.119 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:04.119 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:04.119 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:04.119 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:04.134 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:56:04.197 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:04.190+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:04.258 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:04.258 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:04.274 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:56:04.299 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:04.295+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:04.379 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:56:04.401 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:04.394+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:04.471 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.617954877, 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:56:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:56:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.491907614, 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:56:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:56:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.24470000700001, 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:56:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.216942217, 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:56:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:56:04.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.21682665500001, 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.497 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:56:04.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.502 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.75751931800001, 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.503 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:56:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:56:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.75729326, 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:56:04.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.05069208099999, 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.515 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:56:04.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:56:04.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 116.75238858500001, 2024-09-04T21:56:04.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:04.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:56:04.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:56:04.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 116.57264920599999, 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:56:04.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:56:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.906322301, 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:56:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:56:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:145127 3.19 3.1e5da059 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.011176321000001, 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90329116399999998, 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 145127 2024-09-04T21:56:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458503+0000", 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0029999999999999e-06 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458512+0000", 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3590000000000008e-06 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458530+0000", 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8790000000000001e-05 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458539+0000", 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2290000000000003e-06 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458630+0000", 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0841000000000007e-05 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458644+0000", 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361754+0000", 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90310956399999998 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361791+0000", 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7728999999999997e-05 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4160 3.8 e122/121 3:1505c1f6:::rbd_data.1188b450424.0000000000000003:14 v 122'534, mlcod=0'0)", 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9774962489999997, 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0042635119999999997, 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492182+0000", 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492200+0000", 2024-09-04T21:56:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8009e-05 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492229+0000", 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8469000000000001e-05 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492249+0000", 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9884000000000001e-05 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492350+0000", 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000101572 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492378+0000", 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8138999999999999e-05 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496377+0000", 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0039985480000000002 2024-09-04T21:56:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496444+0000", 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6828000000000005e-05 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4161 3.8 e122/121 3:13882358:::rbd_data.1188b450424.0000000000000004:14 v 122'536, mlcod=0'0)", 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9747121960000005, 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057478820000000002, 2024-09-04T21:56:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494967+0000", 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494982+0000", 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5226e-05 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495010+0000", 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8143000000000001e-05 2024-09-04T21:56:04.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495029+0000", 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8872000000000002e-05 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495238+0000", 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020955799999999999 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495292+0000", 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3548999999999998e-05 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500587+0000", 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052953439999999996 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500712+0000", 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012510000000000001 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4171 3.8 e123/121 3:16896659:::rbd_data.11882296a3bb.00000000000001d8:12 v 123'538, mlcod=122'534)", 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.1065794019999995, 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053660619999999996, 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363098+0000", 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363103+0000", 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9269999999999996e-06 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363114+0000", 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1077000000000001e-05 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363137+0000", 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363420+0000", 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028287199999999998 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363445+0000", 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5471999999999999e-05 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368381+0000", 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0049356000000000001 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368463+0000", 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2495000000000002e-05 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:276 3.1a e123/120 3:58e50b11:::rbd_data.11882296a3bb.000000000000004e:12 v 123'923, mlcod=116'919)", 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.1047933380000003, 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059945689999999999, 2024-09-04T21:56:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364884+0000", 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9599999999999999e-07 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364893+0000", 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6000000000000007e-06 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364919+0000", 2024-09-04T21:56:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6156000000000002e-05 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364936+0000", 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6920000000000001e-05 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365020+0000", 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4146000000000004e-05 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365036+0000", 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6277e-05 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370836+0000", 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058004290000000002 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370878+0000", 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1344999999999998e-05 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146120 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.8232897230000003, 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040719019999999996, 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146120 2024-09-04T21:56:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646390+0000", 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0630000000000002e-06 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646410+0000", 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9749999999999999e-05 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646453+0000", 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2879000000000001e-05 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646476+0000", 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3886000000000001e-05 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646633+0000", 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015649199999999999 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646831+0000", 2024-09-04T21:56:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019817899999999999 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646996+0000", 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000164745 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.648404+0000", 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001408149 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650116+0000", 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001711775 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650329+0000", 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021338100000000001 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650459+0000", 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129603 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146127 3.0 3:04d53328:::rbd_data.1188e6876e06.000000000000000e:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.8144273650000002, 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038955439999999999, 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146127 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655252+0000", 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7269999999999999e-06 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655263+0000", 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0893000000000001e-05 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655294+0000", 2024-09-04T21:56:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1479000000000001e-05 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655310+0000", 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5625999999999999e-05 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655473+0000", 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016301900000000001 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655790+0000", 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000317492 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.656104+0000", 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031342799999999998 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.657718+0000", 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001614571 2024-09-04T21:56:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.658939+0000", 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012200869999999999 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.659027+0000", 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8450999999999997e-05 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.659145+0000", 2024-09-04T21:56:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011777099999999999 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146276 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:head [delete] snapc e=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.7760921139999999, 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051386740000000002, 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146276 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693587+0000", 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5600000000000001e-06 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693595+0000", 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2640000000000008e-06 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693618+0000", 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3051999999999999e-05 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693630+0000", 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2074e-05 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693705+0000", 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4867e-05 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693851+0000", 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014598700000000001 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693977+0000", 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012523100000000001 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.695244+0000", 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001267518 2024-09-04T21:56:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698604+0000", 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003359608 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698671+0000", 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7810000000000003e-05 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698723+0000", 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1703000000000003e-05 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:360 3.1a e131/120 3:5dadb212:::rbd_data.118892c64562.000000000000003d:c v 131'931, mlcod=127'928)", 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.092049008, 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0063556569999999998, 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-04T21:56:04.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377640+0000", 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1776999999999999e-05 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377683+0000", 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2289000000000001e-05 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377705+0000", 2024-09-04T21:56:04.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2245000000000001e-05 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377894+0000", 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018913 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377951+0000", 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7318000000000002e-05 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.383885+0000", 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059336370000000003 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.383983+0000", 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8404000000000002e-05 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:04.585 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:04.587 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:56:04.587 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:04.579+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:04.688 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:04.687+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:04.708 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:04.708 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:04.708 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:04.708 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:04.722 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:56:04.788 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:04.782+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:04.845 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:04.846 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:04.846 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:04.846 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:04.846 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:04.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:56:04.889 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:04.883+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:04.990 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:04.983+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:05.029 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:05.029 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:05.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:05.029 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:05.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.029 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131561 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 835584~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.998577105999999, 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111509741, 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131561 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027841+0000", 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2899999999999999e-06 2024-09-04T21:56:05.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027847+0000", 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8289999999999996e-06 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027868+0000", 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1444999999999999e-05 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027885+0000", 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6149000000000001e-05 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139001+0000", 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111116231 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139139+0000", 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013808700000000001 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139350+0000", 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021070999999999999 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131562 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 925696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.998508878000003, 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111852167, 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131562 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027909+0000", 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.164e-06 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027922+0000", 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2955999999999999e-05 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027942+0000", 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9930999999999999e-05 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027955+0000", 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2916e-05 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139429+0000", 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111473537 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139560+0000", 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013118200000000001 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139760+0000", 2024-09-04T21:56:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020048099999999999 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131563 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 4022272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.998437985000002, 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112167249, 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131563 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028594+0000", 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061550000000000005 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028604+0000", 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3889999999999994e-06 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028627+0000", 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3309e-05 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028644+0000", 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7062e-05 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139847+0000", 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111203128 2024-09-04T21:56:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139975+0000", 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012794799999999999 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140146+0000", 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017091299999999999 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131564 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1015808~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.997742639999998, 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11196729499999999, 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131564 2024-09-04T21:56:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034391+0000", 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005716881 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034402+0000", 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0448999999999999e-05 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034426+0000", 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3975e-05 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034445+0000", 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9386e-05 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140259+0000", 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10581388899999999 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140415+0000", 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015661599999999999 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140642+0000", 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226099 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131577 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2097152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.991452232999997, 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111297805, 2024-09-04T21:56:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131577 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034966+0000", 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034971+0000", 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0200000000000002e-06 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035005+0000", 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3763999999999997e-05 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035023+0000", 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8232000000000001e-05 2024-09-04T21:56:05.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.145896+0000", 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.110872963 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146021+0000", 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124888 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146262+0000", 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024174099999999999 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131578 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2187264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.991370207999999, 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111709911, 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.053 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131578 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:05.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035048+0000", 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3400000000000001e-06 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035054+0000", 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7660000000000004e-06 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035072+0000", 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8224999999999999e-05 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035085+0000", 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3152999999999999e-05 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146367+0000", 2024-09-04T21:56:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111282279 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146521+0000", 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015351399999999999 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146757+0000", 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000235634 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131579 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2277376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.991313339000001, 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11209353800000001, 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131579 2024-09-04T21:56:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035105+0000", 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2470000000000001e-06 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035110+0000", 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.237e-06 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035131+0000", 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1277999999999999e-05 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035153+0000", 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1959000000000001e-05 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146846+0000", 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11169317600000001 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146984+0000", 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137095 2024-09-04T21:56:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147197+0000", 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000213546 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131580 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2367488~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.991244233000003, 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112535119, 2024-09-04T21:56:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131580 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042368+0000", 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071953859999999998 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042379+0000", 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1069e-05 2024-09-04T21:56:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042403+0000", 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4158000000000001e-05 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042422+0000", 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8983000000000001e-05 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147304+0000", 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10488141500000001 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147478+0000", 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386300000000001 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147708+0000", 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023024499999999999 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131596 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 49152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.983347535999997, 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111536857, 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131596 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043071+0000", 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3799999999999999e-06 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043077+0000", 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043096+0000", 2024-09-04T21:56:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9681e-05 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043113+0000", 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6416e-05 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154171+0000", 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111058771 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154353+0000", 2024-09-04T21:56:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018107400000000001 2024-09-04T21:56:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154606+0000", 2024-09-04T21:56:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025366800000000002 2024-09-04T21:56:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131597 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3809280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.983284079999997, 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111867072, 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131597 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043134+0000", 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2529999999999999e-06 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043147+0000", 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2693e-05 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043164+0000", 2024-09-04T21:56:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043177+0000", 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729e-05 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154692+0000", 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111515138 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154825+0000", 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013267800000000001 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155000+0000", 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017526900000000001 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131598 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.983220213999999, 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112278195, 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131598 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050331+0000", 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071345200000000001 2024-09-04T21:56:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050342+0000", 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1189000000000001e-05 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050367+0000", 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4621e-05 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050387+0000", 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819000000000001e-05 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155115+0000", 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104728349 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155274+0000", 2024-09-04T21:56:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015923399999999999 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155475+0000", 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020046300000000001 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131614 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 589824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.975253711000001, 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111567737, 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131614 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051164+0000", 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2160000000000001e-06 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051170+0000", 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2390000000000002e-06 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051187+0000", 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7649e-05 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051200+0000", 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2993000000000001e-05 2024-09-04T21:56:05.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162333+0000", 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111132965 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162500+0000", 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016687900000000001 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162731+0000", 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023079599999999999 2024-09-04T21:56:05.086 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131615 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.975198059, 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11193702, 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.087 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131615 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051220+0000", 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051225+0000", 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1399999999999999e-06 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051242+0000", 2024-09-04T21:56:05.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6988999999999999e-05 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051255+0000", 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3261999999999999e-05 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162787+0000", 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111532033 2024-09-04T21:56:05.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162923+0000", 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013530399999999999 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163156+0000", 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233095 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.091 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131616 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.975143488999997, 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112285617, 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131616 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058214+0000", 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069405279999999996 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058226+0000", 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1732999999999999e-05 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058253+0000", 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7574999999999999e-05 2024-09-04T21:56:05.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058276+0000", 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2334999999999999e-05 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163207+0000", 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104930906 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163336+0000", 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129639 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163559+0000", 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022290100000000001 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131631 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1220608~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.096 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.967514981000001, 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111259443, 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131631 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058903+0000", 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.133e-06 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058909+0000", 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6890000000000001e-06 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058926+0000", 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7166000000000001e-05 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058943+0000", 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7028999999999999e-05 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169875+0000", 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11093157400000001 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169996+0000", 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012140499999999999 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170161+0000", 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016544699999999999 2024-09-04T21:56:05.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131632 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1613824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.967452051999999, 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111614907, 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131632 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058966+0000", 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058974+0000", 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9559999999999997e-06 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059013+0000", 2024-09-04T21:56:05.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8688e-05 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059027+0000", 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170250+0000", 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111222396 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170381+0000", 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013136800000000001 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170580+0000", 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019870700000000001 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131633 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1703936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.105 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.967365037999997, 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11190109299999999, 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131633 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059053+0000", 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.353e-06 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059067+0000", 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3482999999999999e-05 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059104+0000", 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7657999999999998e-05 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059129+0000", 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4380999999999999e-05 2024-09-04T21:56:05.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170639+0000", 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111510427 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170761+0000", 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122302 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170953+0000", 2024-09-04T21:56:05.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019148900000000001 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131634 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.967265118, 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11221152299999999, 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.110 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131634 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059154+0000", 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059160+0000", 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059199+0000", 2024-09-04T21:56:05.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9190999999999998e-05 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059220+0000", 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0499000000000001e-05 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171022+0000", 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11180261599999999 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171158+0000", 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135401 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171363+0000", 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020553999999999999 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131635 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1794048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:05.114 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.967175656999999, 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112472239, 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131635 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:05.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066220+0000", 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069786229999999998 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066229+0000", 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066251+0000", 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1829000000000001e-05 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066273+0000", 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2722e-05 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171426+0000", 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10515273 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171548+0000", 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012224 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171714+0000", 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000165289 2024-09-04T21:56:05.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131652 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2785280~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.959285369, 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111334216, 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:05.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131652 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:05.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074601+0000", 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074699279999999998 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074617+0000", 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5882000000000001e-05 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074651+0000", 2024-09-04T21:56:05.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4100999999999998e-05 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074674+0000", 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2858999999999999e-05 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178090+0000", 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10341562999999999 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:05.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178247+0000", 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156673 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178466+0000", 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021914300000000001 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.123 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:05.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:56:05.125 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:05.123+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:05.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:56:05.226 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:05.223+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:05.295 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:05.296 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:05.296 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:05.296 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:05.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:56:05.325 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:05.325 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:05.326 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:05.319+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:05.343 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:05.344 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:05.363 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:05.388 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:05.388 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:05.414 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:05.414 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:05.428 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:05.422+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:05.431 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-04T21:56:05.431 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:56:05.431 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-04T21:56:05.431 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-04T21:56:05.435 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:05.435 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:05.441 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:05.441 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:05.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:56:05.457 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:05.457 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:05.478 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:05.478 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:05.529 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:05.523+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:05.574 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:05.574 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:05.588 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:56:05.636 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:05.631+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:05.720 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:05.721 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 133, 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "7c7b20bb-5983-49a4-8b2c-296a55ea8d95", 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-04T21:53:25.631463+0000", 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-04T21:56:05.356744+0000", 2024-09-04T21:56:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "last_up_change": "2024-09-04T21:55:37.706985+0000", 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "last_in_change": "2024-09-04T21:56:00.354804+0000", 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_num": 5799936, 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_set": [ 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "pglog_hardlimit", 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "purged_snapdirs", 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_deletes", 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "sortbitwise" 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_version": 7, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "full_ratio": 0.94999998807907104, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_snapshot": "", 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_max": 3, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "max_osd": 8, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "require_min_compat_client": "luminous", 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "min_compat_client": "jewel", 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "require_osd_release": "squid", 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "allow_crimson": false, 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pools": [ 2024-09-04T21:56:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 1, 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_name": ".mgr", 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "create_time": "2024-09-04T21:53:36.459454+0000", 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 1, 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_names": "hashpspool", 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "type": 1, 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 2, 2024-09-04T21:56:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "min_size": 1, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_rule": 0, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_count": 0, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_target": 0, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "is_stretch_pool": false, 2024-09-04T21:56:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "object_hash": 2, 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_autoscale_mode": "off", 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num": 1, 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num": 1, 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num_target": 1, 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_target": 1, 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_pending": 1, 2024-09-04T21:56:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "last_pg_merge_meta": { 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "source_pgid": "0.0", 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "ready_epoch": 0, 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_started": 0, 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_clean": 0, 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "source_version": "0'0", 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "target_version": "0'0" 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "last_change": "17", 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend": "0", 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "auid": 0, 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_mode": "selfmanaged", 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_seq": 0, 2024-09-04T21:56:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_epoch": 0, 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_snaps": [], 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps": "[]", 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_bytes": 0, 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_objects": 0, 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "tiers": [], 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_of": -1, 2024-09-04T21:56:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "read_tier": -1, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "write_tier": -1, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_mode": "none", 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_bytes": 0, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_objects": 0, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T21:56:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_flush_age": 0, 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_evict_age": 0, 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profile": "", 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_params": { 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "none" 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_period": 0, 2024-09-04T21:56:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_count": 0, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "use_gmt_hitset": true, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "min_read_recency_for_promote": 0, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "min_write_recency_for_promote": 0, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_search_last_n": 0, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "grade_table": [], 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "stripe_width": 0, 2024-09-04T21:56:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "expected_num_objects": 0, 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "fast_read": false, 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "options": { 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_max": 32, 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_min": 1 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "application_metadata": { 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": {} 2024-09-04T21:56:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "read_balance": { 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "score_type": "Fair distribution", 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "score_acting": 6.9000000953674316, 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "score_stable": 6.9000000953674316, 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "optimal_score": 0.28999999165534973, 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_acting": 2, 2024-09-04T21:56:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_stable": 2, 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity_weighted": 0.875, 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity": 1, 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 2, 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_name": "rbd", 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "create_time": "2024-09-04T21:53:43.849962+0000", 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 8193, 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "type": 1, 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 2, 2024-09-04T21:56:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "min_size": 1, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_rule": 0, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_count": 0, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_target": 0, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "is_stretch_pool": false, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "object_hash": 2, 2024-09-04T21:56:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_autoscale_mode": "off", 2024-09-04T21:56:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num": 8, 2024-09-04T21:56:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num": 8, 2024-09-04T21:56:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num_target": 8, 2024-09-04T21:56:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_target": 8, 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_pending": 8, 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "last_pg_merge_meta": { 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "source_pgid": "0.0", 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "ready_epoch": 0, 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_started": 0, 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_clean": 0, 2024-09-04T21:56:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: "source_version": "0'0", 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "target_version": "0'0" 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "last_change": "21", 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend": "0", 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "auid": 0, 2024-09-04T21:56:05.750 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_mode": "selfmanaged", 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_seq": 2, 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_epoch": 21, 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_snaps": [], 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps": "[]", 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_bytes": 0, 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_objects": 0, 2024-09-04T21:56:05.751 INFO:teuthology.orchestra.run.smithi112.stdout: "tiers": [], 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_of": -1, 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "read_tier": -1, 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "write_tier": -1, 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_mode": "none", 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_bytes": 0, 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_objects": 0, 2024-09-04T21:56:05.752 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_flush_age": 0, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_evict_age": 0, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profile": "", 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_params": { 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "none" 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_period": 0, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_count": 0, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "use_gmt_hitset": true, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "min_read_recency_for_promote": 0, 2024-09-04T21:56:05.753 INFO:teuthology.orchestra.run.smithi112.stdout: "min_write_recency_for_promote": 0, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_search_last_n": 0, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "grade_table": [], 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "stripe_width": 0, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "expected_num_objects": 0, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "fast_read": false, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "options": {}, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "application_metadata": { 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd": {} 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "read_balance": { 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "score_type": "Fair distribution", 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "score_acting": 2.630000114440918, 2024-09-04T21:56:05.754 INFO:teuthology.orchestra.run.smithi112.stdout: "score_stable": 2.630000114440918, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "optimal_score": 1, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_acting": 2.630000114440918, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_stable": 2.630000114440918, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity_weighted": 0.875, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity": 1, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 3, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_name": "pool_client.0", 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "create_time": "2024-09-04T21:53:54.481634+0000", 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 8193, 2024-09-04T21:56:05.755 INFO:teuthology.orchestra.run.smithi112.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "type": 1, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 2, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "min_size": 1, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_rule": 0, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_count": 0, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_target": 0, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "is_stretch_pool": false, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "object_hash": 2, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_autoscale_mode": "off", 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num": 32, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num": 32, 2024-09-04T21:56:05.756 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_placement_num_target": 32, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_target": 32, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_num_pending": 32, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_pg_merge_meta": { 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "source_pgid": "0.0", 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "ready_epoch": 0, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_started": 0, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_epoch_clean": 0, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "source_version": "0'0", 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "target_version": "0'0" 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_change": "131", 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend": "0", 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T21:56:05.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "auid": 0, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_mode": "selfmanaged", 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_seq": 36, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "snap_epoch": 131, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "pool_snaps": [], 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps": "[]", 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_bytes": 0, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "quota_max_objects": 0, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "tiers": [], 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_of": -1, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "read_tier": -1, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "write_tier": -1, 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_mode": "none", 2024-09-04T21:56:05.758 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_bytes": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "target_max_objects": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_flush_age": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "cache_min_evict_age": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profile": "", 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_params": { 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "none" 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_period": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_count": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "use_gmt_hitset": true, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "min_read_recency_for_promote": 0, 2024-09-04T21:56:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: "min_write_recency_for_promote": 0, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "hit_set_search_last_n": 0, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "grade_table": [], 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "stripe_width": 0, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "expected_num_objects": 0, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "fast_read": false, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "options": {}, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "application_metadata": { 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "rbd": {} 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "read_balance": { 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "score_type": "Fair distribution", 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "score_acting": 1.75, 2024-09-04T21:56:05.760 INFO:teuthology.orchestra.run.smithi112.stdout: "score_stable": 1.75, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "optimal_score": 1.1399999856948853, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_acting": 2, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "raw_score_stable": 2, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity_weighted": 0.875, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity": 1, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "osds": [ 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 0, 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "d210c577-fc3d-48c0-9acb-67afdd5a485a", 2024-09-04T21:56:05.761 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 132, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6816", 2024-09-04T21:56:05.762 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6817", 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6818", 2024-09-04T21:56:05.763 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6819", 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6822", 2024-09-04T21:56:05.764 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6823", 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6820", 2024-09-04T21:56:05.765 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6821", 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3808224845 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6817/3808224845", 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6819/3808224845", 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6823/3808224845", 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6821/3808224845", 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.766 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 1, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "3d1c8d55-e5ac-4787-a725-6895bb2bf3f9", 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 130, 2024-09-04T21:56:05.767 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6824", 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6825", 2024-09-04T21:56:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6826", 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6827", 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6830", 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6831", 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.771 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6828", 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6829", 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 245081828 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6825/245081828", 2024-09-04T21:56:05.772 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6827/245081828", 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6831/245081828", 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6829/245081828", 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 2, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "5a67b7d1-ef0a-4935-9af0-01389e12d43e", 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 14, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 130, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:56:05.774 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6800", 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6801", 2024-09-04T21:56:05.775 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6802", 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6803", 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6806", 2024-09-04T21:56:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6807", 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6804", 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6805", 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 1590148387 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6801/1590148387", 2024-09-04T21:56:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6803/1590148387", 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6807/1590148387", 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6805/1590148387", 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 3, 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "0b3ec943-bdb7-47e3-bcca-b52070713e9c", 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 14, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 132, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6808", 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6809", 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6810", 2024-09-04T21:56:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6811", 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6814", 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6815", 2024-09-04T21:56:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6812", 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6813", 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2461796488 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6809/2461796488", 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.112:6811/2461796488", 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.112:6815/2461796488", 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.112:6813/2461796488", 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 4, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "09964dd2-405d-483d-822d-1b8c74497413", 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 15, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 84, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 105, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 132, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 85, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6824", 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6825", 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6826", 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6827", 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6830", 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6831", 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6828", 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6829", 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3575407730 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6825/3575407730", 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6827/3575407730", 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6831/3575407730", 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6829/3575407730", 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 5, 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "87306896-7b79-487f-ab06-ff337bb72f37", 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.794 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 0, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 13, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 85, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.795 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6807", 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6809", 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.796 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6810", 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6811", 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.797 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6814", 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6815", 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.798 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6812", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6813", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 352715412 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6809/352715412", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6811/352715412", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6815/352715412", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6813/352715412", 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 6, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "e7948c3b-e9b4-4a9b-99be-8364f09b8c1f", 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 0, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 0, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 15, 2024-09-04T21:56:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 132, 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 0, 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6800", 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6801", 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6802", 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6803", 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6806", 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6808", 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6804", 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6805", 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 3465983375 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6801/3465983375", 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6803/3465983375", 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6808/3465983375", 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6805/3465983375", 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 7, 2024-09-04T21:56:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "uuid": "c9fdaad2-f75b-4dcd-a170-3e346fda0e16", 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "up": 1, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "in": 1, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 1, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_affinity": 1, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_begin": 13, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "last_clean_end": 21, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "up_from": 48, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "up_thru": 128, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "down_at": 22, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "lost_at": 0, 2024-09-04T21:56:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6816", 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6817", 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addrs": { 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6818", 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6819", 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addrs": { 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6822", 2024-09-04T21:56:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6823", 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addrs": { 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v2", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6820", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.157:6821", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 471611328 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.157:6817/471611328", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_addr": "172.21.15.157:6819/471611328", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_back_addr": "172.21.15.157:6823/471611328", 2024-09-04T21:56:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "heartbeat_front_addr": "172.21.15.157:6821/471611328", 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "state": [ 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "exists", 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "up" 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_xinfo": [ 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 0, 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.725388+0000", 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 1, 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:35.420675+0000", 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 2, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.389632+0000", 2024-09-04T21:56:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 3, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.611983+0000", 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 4, 2024-09-04T21:56:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.796105+0000", 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 85 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 5, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.383432+0000", 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 6, 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.731214+0000", 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 0 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": 7, 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "down_stamp": "0.000000", 2024-09-04T21:56:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_probability": 0, 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "laggy_interval": 0, 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "features": 4540719139924082687, 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "old_weight": 0, 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "last_purged_snaps_scrub": "2024-09-04T21:53:34.469040+0000", 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "dead_epoch": 22 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_upmap": [], 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_upmap_items": [], 2024-09-04T21:56:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_upmap_primaries": [], 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pg_temp": [], 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "primary_temp": [], 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "blocklist": {}, 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "range_blocklist": {}, 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "erasure_code_profiles": { 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "default": { 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "crush-failure-domain": "osd", 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "k": "2", 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "m": "1", 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "plugin": "jerasure", 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "technique": "reed_sol_van" 2024-09-04T21:56:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_snaps_queue": [ 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 3, 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "snaps": [ 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "begin": 12, 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "length": 5 2024-09-04T21:56:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "new_removed_snaps": [], 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "new_purged_snaps": [ 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 3, 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "snaps": [ 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "begin": 17, 2024-09-04T21:56:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "length": 4 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_node_flags": {}, 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "device_class_flags": {}, 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": { 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode_enabled": false, 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_bucket_count": 0, 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "degraded_stretch_mode": 0, 2024-09-04T21:56:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "recovering_stretch_mode": 0, 2024-09-04T21:56:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode_bucket": 0 2024-09-04T21:56:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:05.824 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:05.825 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:05.727+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:05.827 INFO:tasks.thrashosds.thrasher:j is {'epoch': 133, 'fsid': '7c7b20bb-5983-49a4-8b2c-296a55ea8d95', 'created': '2024-09-04T21:53:25.631463+0000', 'modified': '2024-09-04T21:56:05.356744+0000', 'last_up_change': '2024-09-04T21:55:37.706985+0000', 'last_in_change': '2024-09-04T21:56:00.354804+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-04T21:53:36.459454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-04T21:53:43.849962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-04T21:53:54.481634+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 36, 'snap_epoch': 131, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'd210c577-fc3d-48c0-9acb-67afdd5a485a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6816', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6817', 'nonce': 3808224845}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6818', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6819', 'nonce': 3808224845}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6822', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6823', 'nonce': 3808224845}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6820', 'nonce': 3808224845}, {'type': 'v1', 'addr': '172.21.15.112:6821', 'nonce': 3808224845}]}, 'public_addr': '172.21.15.112:6817/3808224845', 'cluster_addr': '172.21.15.112:6819/3808224845', 'heartbeat_back_addr': '172.21.15.112:6823/3808224845', 'heartbeat_front_addr': '172.21.15.112:6821/3808224845', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3d1c8d55-e5ac-4787-a725-6895bb2bf3f9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 245081828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 245081828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6830', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6831', 'nonce': 245081828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6828', 'nonce': 245081828}, {'type': 'v1', 'addr': '172.21.15.112:6829', 'nonce': 245081828}]}, 'public_addr': '172.21.15.112:6825/245081828', 'cluster_addr': '172.21.15.112:6827/245081828', 'heartbeat_back_addr': '172.21.15.112:6831/245081828', 'heartbeat_front_addr': '172.21.15.112:6829/245081828', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5a67b7d1-ef0a-4935-9af0-01389e12d43e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6800', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6801', 'nonce': 1590148387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6802', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6803', 'nonce': 1590148387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6806', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6807', 'nonce': 1590148387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6804', 'nonce': 1590148387}, {'type': 'v1', 'addr': '172.21.15.112:6805', 'nonce': 1590148387}]}, 'public_addr': '172.21.15.112:6801/1590148387', 'cluster_addr': '172.21.15.112:6803/1590148387', 'heartbeat_back_addr': '172.21.15.112:6807/1590148387', 'heartbeat_front_addr': '172.21.15.112:6805/1590148387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0b3ec943-bdb7-47e3-bcca-b52070713e9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6808', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6809', 'nonce': 2461796488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6810', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6811', 'nonce': 2461796488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6814', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6815', 'nonce': 2461796488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6812', 'nonce': 2461796488}, {'type': 'v1', 'addr': '172.21.15.112:6813', 'nonce': 2461796488}]}, 'public_addr': '172.21.15.112:6809/2461796488', 'cluster_addr': '172.21.15.112:6811/2461796488', 'heartbeat_back_addr': '172.21.15.112:6815/2461796488', 'heartbeat_front_addr': '172.21.15.112:6813/2461796488', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '09964dd2-405d-483d-822d-1b8c74497413', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 15, 'last_clean_end': 84, 'up_from': 105, 'up_thru': 132, 'down_at': 85, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6824', 'nonce': 3575407730}, {'type': 'v1', 'addr': '172.21.15.157:6825', 'nonce': 3575407730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6826', 'nonce': 3575407730}, {'type': 'v1', 'addr': '172.21.15.157:6827', 'nonce': 3575407730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6830', 'nonce': 3575407730}, {'type': 'v1', 'addr': '172.21.15.157:6831', 'nonce': 3575407730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6828', 'nonce': 3575407730}, {'type': 'v1', 'addr': '172.21.15.157:6829', 'nonce': 3575407730}]}, 'public_addr': '172.21.15.157:6825/3575407730', 'cluster_addr': '172.21.15.157:6827/3575407730', 'heartbeat_back_addr': '172.21.15.157:6831/3575407730', 'heartbeat_front_addr': '172.21.15.157:6829/3575407730', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '87306896-7b79-487f-ab06-ff337bb72f37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 85, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6807', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6809', 'nonce': 352715412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6810', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6811', 'nonce': 352715412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6814', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6815', 'nonce': 352715412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6812', 'nonce': 352715412}, {'type': 'v1', 'addr': '172.21.15.157:6813', 'nonce': 352715412}]}, 'public_addr': '172.21.15.157:6809/352715412', 'cluster_addr': '172.21.15.157:6811/352715412', 'heartbeat_back_addr': '172.21.15.157:6815/352715412', 'heartbeat_front_addr': '172.21.15.157:6813/352715412', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7948c3b-e9b4-4a9b-99be-8364f09b8c1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6800', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6801', 'nonce': 3465983375}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6802', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6803', 'nonce': 3465983375}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6806', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6808', 'nonce': 3465983375}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6804', 'nonce': 3465983375}, {'type': 'v1', 'addr': '172.21.15.157:6805', 'nonce': 3465983375}]}, 'public_addr': '172.21.15.157:6801/3465983375', 'cluster_addr': '172.21.15.157:6803/3465983375', 'heartbeat_back_addr': '172.21.15.157:6808/3465983375', 'heartbeat_front_addr': '172.21.15.157:6805/3465983375', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9fdaad2-f75b-4dcd-a170-3e346fda0e16', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 21, 'up_from': 48, 'up_thru': 128, 'down_at': 22, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6816', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6817', 'nonce': 471611328}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6818', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6819', 'nonce': 471611328}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6822', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6823', 'nonce': 471611328}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6820', 'nonce': 471611328}, {'type': 'v1', 'addr': '172.21.15.157:6821', 'nonce': 471611328}]}, 'public_addr': '172.21.15.157:6817/471611328', 'cluster_addr': '172.21.15.157:6819/471611328', 'heartbeat_back_addr': '172.21.15.157:6823/471611328', 'heartbeat_front_addr': '172.21.15.157:6821/471611328', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.725388+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:35.420675+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.389632+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.611983+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.796105+0000', 'dead_epoch': 85}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.383432+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.731214+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-04T21:53:34.469040+0000', 'dead_epoch': 22}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 12, 'length': 5}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 17, 'length': 4}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-04T21:56:05.827 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:05.827 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:56:05.868 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:05.863+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:05.934 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:05.927+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:05.993 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:05.993 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.008 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:56:06.037 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:06.030+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:06.135 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:06.137 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:06.131+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:06.152 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-04T21:56:05.360257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347766063,"num_objects":549,"num_object_clones":112,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":177,"num_objects_misplaced":215,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":296,"num_read":189717,"num_read_kb":671667,"num_write":20343,"num_write_kb":315914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2357,"num_bytes_recovered":83339011,"num_keys_recovered":295,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16812,"ondisk_log_size":16812,"up":82,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":327040,"kb_used_data":25668,"kb_used_omap":46,"kb_used_meta":300881,"kb_avail":754647680,"statfs":{"total":773094113280,"available":772759224320,"internally_reserved":0,"allocated":26284032,"data_stored":131837950,"data_compressed":7481571,"data_compressed_allocated":11087872,"data_compressed_original":124099321,"omap_allocated":47982,"internal_metadata":308102290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-103877687,"num_objects":-50,"num_object_clones":-15,"num_object_copies":-100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":177,"num_objects_misplaced":215,"num_objects_unfound":0,"num_objects_dirty":-50,"num_whiteouts":-5,"num_read":653,"num_read_kb":561,"num_write":105,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7444072,"num_keys_recovered":0,"num_objects_omap":-3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.992757"},"pg_stats":[{"pgid":"3.1f","version":"130'761","reported_seq":2872,"reported_epoch":131,"state":"creating+peering","last_fresh":"2024-09-04T21:56:03.340383+0000","last_change":"2024-09-04T21:56:03.720627+0000","last_active":"2024-09-04T21:56:03.720627+0000","last_peered":"2024-09-04T21:56:03.340383+0000","last_clean":"2024-09-04T21:55:59.817643+0000","last_became_active":"2024-09-04T21:56:02.684520+0000","last_became_peered":"2024-09-04T21:56:02.684520+0000","last_unstale":"2024-09-04T21:56:03.340383+0000","last_undegraded":"2024-09-04T21:56:03.340383+0000","last_fullsized":"2024-09-04T21:56:03.340383+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070464599999999996,"stat_sum":{"num_bytes":14037008,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":9,"num_read":5090,"num_read_kb":7783,"num_write":1084,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.1e","version":"126'370","reported_seq":3416,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.372587+0000","last_change":"2024-09-04T21:56:00.369531+0000","last_active":"2024-09-04T21:56:02.372587+0000","last_peered":"2024-09-04T21:56:02.372587+0000","last_clean":"2024-09-04T21:56:02.372587+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:02.372587+0000","last_undegraded":"2024-09-04T21:56:02.372587+0000","last_fullsized":"2024-09-04T21:56:02.372587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:57:14.091803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013128669999999999,"stat_sum":{"num_bytes":5741119,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":11,"num_read":3612,"num_read_kb":13268,"num_write":343,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.1d","version":"126'541","reported_seq":19021,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-04T21:56:01.370738+0000","last_change":"2024-09-04T21:56:00.384279+0000","last_active":"2024-09-04T21:56:01.370738+0000","last_peered":"2024-09-04T21:56:01.370738+0000","last_clean":"2024-09-04T21:56:01.370738+0000","last_became_active":"2024-09-04T21:55:38.745373+0000","last_became_peered":"2024-09-04T21:55:38.745373+0000","last_unstale":"2024-09-04T21:56:01.370738+0000","last_undegraded":"2024-09-04T21:56:01.370738+0000","last_fullsized":"2024-09-04T21:56:01.370738+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:32:53.903463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075672090000000001,"stat_sum":{"num_bytes":25821648,"num_objects":18,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":8,"num_read":19910,"num_read_kb":23225,"num_write":715,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":911480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.1c","version":"128'373","reported_seq":2317,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-04T21:56:01.370074+0000","last_change":"2024-09-04T21:56:00.387974+0000","last_active":"2024-09-04T21:56:01.370074+0000","last_peered":"2024-09-04T21:56:01.370074+0000","last_clean":"2024-09-04T21:56:01.370074+0000","last_became_active":"2024-09-04T21:55:38.745462+0000","last_became_peered":"2024-09-04T21:55:38.745462+0000","last_unstale":"2024-09-04T21:56:01.370074+0000","last_undegraded":"2024-09-04T21:56:01.370074+0000","last_fullsized":"2024-09-04T21:56:01.370074+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T21:59:36.805490+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.012448846,"stat_sum":{"num_bytes":8345209,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":9,"num_read":2155,"num_read_kb":21076,"num_write":365,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.1b","version":"125'279","reported_seq":1581,"reported_epoch":131,"state":"peering","last_fresh":"2024-09-04T21:56:03.135516+0000","last_change":"2024-09-04T21:56:03.720606+0000","last_active":"2024-09-04T21:56:03.720606+0000","last_peered":"2024-09-04T21:56:03.135516+0000","last_clean":"2024-09-04T21:55:59.791776+0000","last_became_active":"2024-09-04T21:56:02.682099+0000","last_became_peered":"2024-09-04T21:56:02.682099+0000","last_unstale":"2024-09-04T21:56:03.135516+0000","last_undegraded":"2024-09-04T21:56:03.135516+0000","last_fullsized":"2024-09-04T21:56:02.373998+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031796369999999999,"stat_sum":{"num_bytes":4664154,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":9,"num_read":2265,"num_read_kb":17539,"num_write":266,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.1a","version":"127'929","reported_seq":17371,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-04T21:56:01.369576+0000","last_change":"2024-09-04T21:56:00.386846+0000","last_active":"2024-09-04T21:56:01.369576+0000","last_peered":"2024-09-04T21:56:01.369576+0000","last_clean":"2024-09-04T21:56:01.369576+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:01.369576+0000","last_undegraded":"2024-09-04T21:56:01.369576+0000","last_fullsized":"2024-09-04T21:56:01.369576+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:53:00.706216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093653179999999992,"stat_sum":{"num_bytes":16187860,"num_objects":24,"num_object_clones":6,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":12,"num_read":19860,"num_read_kb":57087,"num_write":1204,"num_write_kb":17248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.19","version":"131'1210","reported_seq":3182,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.397438+0000","last_change":"2024-09-04T21:56:03.397880+0000","last_active":"2024-09-04T21:56:03.397438+0000","last_peered":"2024-09-04T21:56:03.397438+0000","last_clean":"2024-09-04T21:56:03.397438+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:03.397438+0000","last_undegraded":"2024-09-04T21:56:03.397438+0000","last_fullsized":"2024-09-04T21:56:03.397438+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1210,"log_dups_size":0,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:49:59.956920+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.015643491999999998,"stat_sum":{"num_bytes":201038,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":7,"num_read":6859,"num_read_kb":17210,"num_write":1893,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.18","version":"127'370","reported_seq":4360,"reported_epoch":130,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-04T21:56:02.686571+0000","last_change":"2024-09-04T21:56:02.686845+0000","last_active":"2024-09-04T21:56:02.686571+0000","last_peered":"2024-09-04T21:56:02.686571+0000","last_clean":"2024-09-04T21:55:59.768347+0000","last_became_active":"2024-09-04T21:56:02.685340+0000","last_became_peered":"2024-09-04T21:56:02.685340+0000","last_unstale":"2024-09-04T21:56:02.686571+0000","last_undegraded":"2024-09-04T21:56:02.376559+0000","last_fullsized":"2024-09-04T21:56:02.372369+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019991449999999999,"stat_sum":{"num_bytes":10982649,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":7,"num_read":4942,"num_read_kb":26011,"num_write":370,"num_write_kb":8119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":7264422,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.17","version":"127'913","reported_seq":3070,"reported_epoch":130,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-04T21:56:02.690910+0000","last_change":"2024-09-04T21:56:02.691254+0000","last_active":"2024-09-04T21:56:02.690910+0000","last_peered":"2024-09-04T21:56:02.690910+0000","last_clean":"2024-09-04T21:55:59.794649+0000","last_became_active":"2024-09-04T21:56:02.689127+0000","last_became_peered":"2024-09-04T21:56:02.689127+0000","last_unstale":"2024-09-04T21:56:02.690910+0000","last_undegraded":"2024-09-04T21:56:02.690910+0000","last_fullsized":"2024-09-04T21:56:02.690910+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":913,"log_dups_size":0,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033872899999999998,"stat_sum":{"num_bytes":19233159,"num_objects":24,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":14,"num_read":5664,"num_read_kb":22372,"num_write":1278,"num_write_kb":7107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3732324,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.16","version":"131'355","reported_seq":2631,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.377771+0000","last_change":"2024-09-04T21:56:03.394350+0000","last_active":"2024-09-04T21:56:04.377771+0000","last_peered":"2024-09-04T21:56:04.377771+0000","last_clean":"2024-09-04T21:56:04.377771+0000","last_became_active":"2024-09-04T21:54:42.679389+0000","last_became_peered":"2024-09-04T21:54:42.679389+0000","last_unstale":"2024-09-04T21:56:04.377771+0000","last_undegraded":"2024-09-04T21:56:04.377771+0000","last_fullsized":"2024-09-04T21:56:04.377771+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:37:46.044904+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0082853070000000004,"stat_sum":{"num_bytes":1083392,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":329,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.15","version":"131'378","reported_seq":3058,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.378193+0000","last_change":"2024-09-04T21:56:03.408613+0000","last_active":"2024-09-04T21:56:04.378193+0000","last_peered":"2024-09-04T21:56:04.378193+0000","last_clean":"2024-09-04T21:56:04.378193+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:04.378193+0000","last_undegraded":"2024-09-04T21:56:04.378193+0000","last_fullsized":"2024-09-04T21:56:04.378193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:25:39.296598+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.021570672999999999,"stat_sum":{"num_bytes":5973495,"num_objects":28,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":19,"num_read":3202,"num_read_kb":27791,"num_write":391,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.14","version":"124'340","reported_seq":1957,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-04T21:56:01.370026+0000","last_change":"2024-09-04T21:56:00.381801+0000","last_active":"2024-09-04T21:56:01.370026+0000","last_peered":"2024-09-04T21:56:01.370026+0000","last_clean":"2024-09-04T21:56:01.370026+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:01.370026+0000","last_undegraded":"2024-09-04T21:56:01.370026+0000","last_fullsized":"2024-09-04T21:56:01.370026+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:50:14.513255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064304330000000002,"stat_sum":{"num_bytes":824054,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":4,"num_read":1593,"num_read_kb":6399,"num_write":362,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"2.7","version":"0'0","reported_seq":249,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.383193+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:03.383193+0000","last_peered":"2024-09-04T21:56:03.383193+0000","last_clean":"2024-09-04T21:56:03.383193+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:03.383193+0000","last_undegraded":"2024-09-04T21:56:03.383193+0000","last_fullsized":"2024-09-04T21:56:03.383193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"130'446","reported_seq":2415,"reported_epoch":131,"state":"peering","last_fresh":"2024-09-04T21:56:02.847759+0000","last_change":"2024-09-04T21:56:03.720617+0000","last_active":"2024-09-04T21:56:03.720617+0000","last_peered":"2024-09-04T21:56:02.847759+0000","last_clean":"2024-09-04T21:55:59.724323+0000","last_became_active":"2024-09-04T21:56:02.665719+0000","last_became_peered":"2024-09-04T21:56:02.665719+0000","last_unstale":"2024-09-04T21:56:02.847759+0000","last_undegraded":"2024-09-04T21:56:02.847759+0000","last_fullsized":"2024-09-04T21:56:02.378865+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012092369999999999,"stat_sum":{"num_bytes":4021866,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":8,"num_read":3307,"num_read_kb":5460,"num_write":389,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.383418+0000","last_change":"2024-09-04T21:55:50.373208+0000","last_active":"2024-09-04T21:56:03.383418+0000","last_peered":"2024-09-04T21:56:03.383418+0000","last_clean":"2024-09-04T21:56:03.383418+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:03.383418+0000","last_undegraded":"2024-09-04T21:56:03.383418+0000","last_fullsized":"2024-09-04T21:56:03.383418+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:34.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"125'317","reported_seq":3336,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.373855+0000","last_change":"2024-09-04T21:56:00.370780+0000","last_active":"2024-09-04T21:56:02.373855+0000","last_peered":"2024-09-04T21:56:02.373855+0000","last_clean":"2024-09-04T21:56:02.373855+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:02.373855+0000","last_undegraded":"2024-09-04T21:56:02.373855+0000","last_fullsized":"2024-09-04T21:56:02.373855+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:19:45.325216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001674839,"stat_sum":{"num_bytes":3168324,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":7,"num_read":3274,"num_read_kb":21958,"num_write":337,"num_write_kb":14374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"2.5","version":"0'0","reported_seq":218,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.378381+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:04.378381+0000","last_peered":"2024-09-04T21:56:04.378381+0000","last_clean":"2024-09-04T21:56:04.378381+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:04.378381+0000","last_undegraded":"2024-09-04T21:56:04.378381+0000","last_fullsized":"2024-09-04T21:56:04.378381+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"130'300","reported_seq":1747,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.396013+0000","last_change":"2024-09-04T21:56:03.397678+0000","last_active":"2024-09-04T21:56:03.396013+0000","last_peered":"2024-09-04T21:56:03.396013+0000","last_clean":"2024-09-04T21:56:03.396013+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:03.396013+0000","last_undegraded":"2024-09-04T21:56:03.396013+0000","last_fullsized":"2024-09-04T21:56:03.396013+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:41:50.758130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014656916000000001,"stat_sum":{"num_bytes":2519588,"num_objects":14,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":8,"num_read":1746,"num_read_kb":7802,"num_write":266,"num_write_kb":3585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.382469+0000","last_change":"2024-09-04T21:56:01.380611+0000","last_active":"2024-09-04T21:56:03.382469+0000","last_peered":"2024-09-04T21:56:03.382469+0000","last_clean":"2024-09-04T21:56:03.382469+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:03.382469+0000","last_undegraded":"2024-09-04T21:56:03.382469+0000","last_fullsized":"2024-09-04T21:56:03.382469+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:44.108771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"125'517","reported_seq":11688,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.372697+0000","last_change":"2024-09-04T21:56:00.372222+0000","last_active":"2024-09-04T21:56:02.372697+0000","last_peered":"2024-09-04T21:56:02.372697+0000","last_clean":"2024-09-04T21:56:02.372697+0000","last_became_active":"2024-09-04T21:54:42.679683+0000","last_became_peered":"2024-09-04T21:54:42.679683+0000","last_unstale":"2024-09-04T21:56:02.372697+0000","last_undegraded":"2024-09-04T21:56:02.372697+0000","last_fullsized":"2024-09-04T21:56:02.372697+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:51:11.981694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003510988,"stat_sum":{"num_bytes":18055236,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":6,"num_read":12464,"num_read_kb":25171,"num_write":564,"num_write_kb":14138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.3","version":"127'455","reported_seq":3718,"reported_epoch":128,"state":"remapped+peering","last_fresh":"2024-09-04T21:56:00.380561+0000","last_change":"2024-09-04T21:56:01.373138+0000","last_active":"2024-09-04T21:56:00.380561+0000","last_peered":"2024-09-04T21:55:59.727782+0000","last_clean":"2024-09-04T21:55:59.727782+0000","last_became_active":"2024-09-04T21:55:41.371293+0000","last_became_peered":"2024-09-04T21:55:41.371293+0000","last_unstale":"2024-09-04T21:56:00.380561+0000","last_undegraded":"2024-09-04T21:56:00.380561+0000","last_fullsized":"2024-09-04T21:56:00.380561+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046612620000000002,"stat_sum":{"num_bytes":1452463,"num_objects":16,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":3299,"num_read_kb":16272,"num_write":503,"num_write_kb":12375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":2080930,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"8"}]},{"pgid":"2.2","version":"21'2","reported_seq":41,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.384090+0000","last_change":"2024-09-04T21:55:50.378307+0000","last_active":"2024-09-04T21:56:03.384090+0000","last_peered":"2024-09-04T21:56:03.384090+0000","last_clean":"2024-09-04T21:56:03.384090+0000","last_became_active":"2024-09-04T21:55:50.371992+0000","last_became_peered":"2024-09-04T21:55:50.371992+0000","last_unstale":"2024-09-04T21:56:03.384090+0000","last_undegraded":"2024-09-04T21:56:03.384090+0000","last_fullsized":"2024-09-04T21:56:03.384090+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:09:28.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"131'344","reported_seq":3069,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.395181+0000","last_change":"2024-09-04T21:56:03.395845+0000","last_active":"2024-09-04T21:56:03.395181+0000","last_peered":"2024-09-04T21:56:03.395181+0000","last_clean":"2024-09-04T21:56:03.395181+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:03.395181+0000","last_undegraded":"2024-09-04T21:56:03.395181+0000","last_fullsized":"2024-09-04T21:56:03.395181+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:00:12.534452+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013225608999999999,"stat_sum":{"num_bytes":2036573,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":11,"num_read":3469,"num_read_kb":53656,"num_write":565,"num_write_kb":21343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"2.1","version":"0'0","reported_seq":238,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.374255+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:56:02.374255+0000","last_peered":"2024-09-04T21:56:02.374255+0000","last_clean":"2024-09-04T21:56:02.374255+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:02.374255+0000","last_undegraded":"2024-09-04T21:56:02.374255+0000","last_fullsized":"2024-09-04T21:56:02.374255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'593","reported_seq":4546,"reported_epoch":131,"state":"peering","last_fresh":"2024-09-04T21:56:02.979422+0000","last_change":"2024-09-04T21:56:03.720586+0000","last_active":"2024-09-04T21:56:03.720586+0000","last_peered":"2024-09-04T21:56:02.979422+0000","last_clean":"2024-09-04T21:55:58.732554+0000","last_became_active":"2024-09-04T21:56:02.674381+0000","last_became_peered":"2024-09-04T21:56:02.674381+0000","last_unstale":"2024-09-04T21:56:02.979422+0000","last_undegraded":"2024-09-04T21:56:02.979422+0000","last_fullsized":"2024-09-04T21:56:02.979422+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931133,"stat_sum":{"num_bytes":36005003,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":9,"num_read":6180,"num_read_kb":34254,"num_write":845,"num_write_kb":14489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"2.0","version":"0'0","reported_seq":206,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.377375+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:04.377375+0000","last_peered":"2024-09-04T21:56:04.377375+0000","last_clean":"2024-09-04T21:56:04.377375+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:04.377375+0000","last_undegraded":"2024-09-04T21:56:04.377375+0000","last_fullsized":"2024-09-04T21:56:04.377375+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"124'384","reported_seq":3428,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.374367+0000","last_change":"2024-09-04T21:56:00.369713+0000","last_active":"2024-09-04T21:56:02.374367+0000","last_peered":"2024-09-04T21:56:02.374367+0000","last_clean":"2024-09-04T21:56:02.374367+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:02.374367+0000","last_undegraded":"2024-09-04T21:56:02.374367+0000","last_fullsized":"2024-09-04T21:56:02.374367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:33:19.960625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011524930000000001,"stat_sum":{"num_bytes":24497962,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":13,"num_read":3779,"num_read_kb":14901,"num_write":354,"num_write_kb":5843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"2.3","version":"19'1","reported_seq":160,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.374524+0000","last_change":"2024-09-04T21:54:42.676579+0000","last_active":"2024-09-04T21:56:02.374524+0000","last_peered":"2024-09-04T21:56:02.374524+0000","last_clean":"2024-09-04T21:56:02.374524+0000","last_became_active":"2024-09-04T21:54:42.676301+0000","last_became_peered":"2024-09-04T21:54:42.676301+0000","last_unstale":"2024-09-04T21:56:02.374524+0000","last_undegraded":"2024-09-04T21:56:02.374524+0000","last_fullsized":"2024-09-04T21:56:02.374524+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:42:21.618311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":302,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.377370+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:04.377370+0000","last_peered":"2024-09-04T21:56:04.377370+0000","last_clean":"2024-09-04T21:56:04.377370+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:04.377370+0000","last_undegraded":"2024-09-04T21:56:04.377370+0000","last_fullsized":"2024-09-04T21:56:04.377370+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"128'542","reported_seq":3586,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.372242+0000","last_change":"2024-09-04T21:56:00.373770+0000","last_active":"2024-09-04T21:56:02.372242+0000","last_peered":"2024-09-04T21:56:02.372242+0000","last_clean":"2024-09-04T21:56:02.372242+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:02.372242+0000","last_undegraded":"2024-09-04T21:56:02.372242+0000","last_fullsized":"2024-09-04T21:56:02.372242+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:28:32.776451+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0031846539999999999,"stat_sum":{"num_bytes":16594312,"num_objects":21,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":9,"num_read":4219,"num_read_kb":15007,"num_write":647,"num_write_kb":7058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.9","version":"126'336","reported_seq":2082,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-04T21:56:01.369753+0000","last_change":"2024-09-04T21:56:00.388333+0000","last_active":"2024-09-04T21:56:01.369753+0000","last_peered":"2024-09-04T21:56:01.369753+0000","last_clean":"2024-09-04T21:56:01.369753+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:01.369753+0000","last_undegraded":"2024-09-04T21:56:01.369753+0000","last_fullsized":"2024-09-04T21:56:01.369753+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:58:43.521500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075779469999999998,"stat_sum":{"num_bytes":2229519,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":7,"num_read":2780,"num_read_kb":16422,"num_write":303,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.a","version":"131'640","reported_seq":2669,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-04T21:56:03.391714+0000","last_change":"2024-09-04T21:56:03.392109+0000","last_active":"2024-09-04T21:56:03.391714+0000","last_peered":"2024-09-04T21:56:03.391714+0000","last_clean":"2024-09-04T21:56:03.391714+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:03.391714+0000","last_undegraded":"2024-09-04T21:56:03.391714+0000","last_fullsized":"2024-09-04T21:56:03.391714+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:53:22.102275+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0088836360000000003,"stat_sum":{"num_bytes":45623729,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":6,"num_read":3587,"num_read_kb":13322,"num_write":778,"num_write_kb":9490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.b","version":"126'360","reported_seq":2130,"reported_epoch":130,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-04T21:56:02.687971+0000","last_change":"2024-09-04T21:56:02.688312+0000","last_active":"2024-09-04T21:56:02.687971+0000","last_peered":"2024-09-04T21:56:02.687971+0000","last_clean":"2024-09-04T21:55:59.548123+0000","last_became_active":"2024-09-04T21:56:02.686876+0000","last_became_peered":"2024-09-04T21:56:02.686876+0000","last_unstale":"2024-09-04T21:56:02.687971+0000","last_undegraded":"2024-09-04T21:56:02.383997+0000","last_fullsized":"2024-09-04T21:56:02.377811+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417668,"stat_sum":{"num_bytes":3571764,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":9,"num_read":2887,"num_read_kb":7865,"num_write":413,"num_write_kb":6205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":1821796,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.c","version":"123'342","reported_seq":3485,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-04T21:56:02.373094+0000","last_change":"2024-09-04T21:56:00.371189+0000","last_active":"2024-09-04T21:56:02.373094+0000","last_peered":"2024-09-04T21:56:02.373094+0000","last_clean":"2024-09-04T21:56:02.373094+0000","last_became_active":"2024-09-04T21:54:42.679512+0000","last_became_peered":"2024-09-04T21:54:42.679512+0000","last_unstale":"2024-09-04T21:56:02.373094+0000","last_undegraded":"2024-09-04T21:56:02.373094+0000","last_fullsized":"2024-09-04T21:56:02.373094+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:29:43.472835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029664460000000002,"stat_sum":{"num_bytes":31270327,"num_objects":19,"num_object_clones":5,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":9,"num_read":3701,"num_read_kb":12906,"num_write":310,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"8"}]},{"pgid":"3.d","version":"131'409","reported_seq":11110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.377873+0000","last_change":"2024-09-04T21:56:03.399822+0000","last_active":"2024-09-04T21:56:04.377873+0000","last_peered":"2024-09-04T21:56:04.377873+0000","last_clean":"2024-09-04T21:56:04.377873+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:04.377873+0000","last_undegraded":"2024-09-04T21:56:04.377873+0000","last_fullsized":"2024-09-04T21:56:04.377873+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:51:54.905408+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013605173999999999,"stat_sum":{"num_bytes":2824364,"num_objects":20,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":14,"num_read":11855,"num_read_kb":18507,"num_write":391,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.e","version":"130'384","reported_seq":1281,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.378061+0000","last_change":"2024-09-04T21:56:03.400835+0000","last_active":"2024-09-04T21:56:04.378061+0000","last_peered":"2024-09-04T21:56:04.378061+0000","last_clean":"2024-09-04T21:56:04.378061+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:04.378061+0000","last_undegraded":"2024-09-04T21:56:04.378061+0000","last_fullsized":"2024-09-04T21:56:04.378061+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:29:19.112446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013898343000000001,"stat_sum":{"num_bytes":7043595,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":9,"num_read":2160,"num_read_kb":12559,"num_write":458,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.f","version":"131'479","reported_seq":11842,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.378525+0000","last_change":"2024-09-04T21:56:03.407811+0000","last_active":"2024-09-04T21:56:04.378525+0000","last_peered":"2024-09-04T21:56:04.378525+0000","last_clean":"2024-09-04T21:56:04.378525+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:04.378525+0000","last_undegraded":"2024-09-04T21:56:04.378525+0000","last_fullsized":"2024-09-04T21:56:04.378525+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:34:42.666495+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.020020348,"stat_sum":{"num_bytes":3440656,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":13,"num_read":11626,"num_read_kb":24784,"num_write":463,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.10","version":"125'806","reported_seq":3566,"reported_epoch":131,"state":"active+undersized+remapped","last_fresh":"2024-09-04T21:56:03.624072+0000","last_change":"2024-09-04T21:56:03.624629+0000","last_active":"2024-09-04T21:56:03.624072+0000","last_peered":"2024-09-04T21:56:03.624072+0000","last_clean":"2024-09-04T21:55:59.526537+0000","last_became_active":"2024-09-04T21:56:02.690018+0000","last_became_peered":"2024-09-04T21:56:02.690018+0000","last_unstale":"2024-09-04T21:56:03.624072+0000","last_undegraded":"2024-09-04T21:56:03.624072+0000","last_fullsized":"2024-09-04T21:56:02.377458+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019631340000000001,"stat_sum":{"num_bytes":1735917,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":13,"num_read":5301,"num_read_kb":12768,"num_write":1014,"num_write_kb":8249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.11","version":"131'547","reported_seq":5852,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.377419+0000","last_change":"2024-09-04T21:56:03.399474+0000","last_active":"2024-09-04T21:56:04.377419+0000","last_peered":"2024-09-04T21:56:04.377419+0000","last_clean":"2024-09-04T21:56:04.377419+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:04.377419+0000","last_undegraded":"2024-09-04T21:56:04.377419+0000","last_fullsized":"2024-09-04T21:56:04.377419+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:10:34.806935+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01323452,"stat_sum":{"num_bytes":9092031,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":6,"num_read":6655,"num_read_kb":44034,"num_write":659,"num_write_kb":25331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]},{"pgid":"3.12","version":"126'748","reported_seq":13510,"reported_epoch":130,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-04T21:56:02.683061+0000","last_change":"2024-09-04T21:56:02.683061+0000","last_active":"2024-09-04T21:56:02.683061+0000","last_peered":"2024-09-04T21:56:02.683061+0000","last_clean":"2024-09-04T21:55:58.522674+0000","last_became_active":"2024-09-04T21:56:02.679860+0000","last_became_peered":"2024-09-04T21:56:02.679860+0000","last_unstale":"2024-09-04T21:56:02.683061+0000","last_undegraded":"2024-09-04T21:56:02.683061+0000","last_fullsized":"2024-09-04T21:56:02.683061+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087600199999999997,"stat_sum":{"num_bytes":18635454,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":121,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":8,"num_read":14612,"num_read_kb":38088,"num_write":910,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":7361343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[{"start":"e","length":"7"}]},{"pgid":"3.13","version":"130'969","reported_seq":2540,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-04T21:56:04.378090+0000","last_change":"2024-09-04T21:56:03.402735+0000","last_active":"2024-09-04T21:56:04.378090+0000","last_peered":"2024-09-04T21:56:04.378090+0000","last_clean":"2024-09-04T21:56:04.378090+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:04.378090+0000","last_undegraded":"2024-09-04T21:56:04.378090+0000","last_fullsized":"2024-09-04T21:56:04.378090+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:42:10.009797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015525524000000001,"stat_sum":{"num_bytes":262208,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":7,"num_read":4701,"num_read_kb":19306,"num_write":1471,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"9"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":347175676,"num_objects":545,"num_object_clones":112,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":177,"num_objects_misplaced":215,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":296,"num_read":189651,"num_read_kb":671612,"num_write":20240,"num_write_kb":314284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":82158237,"num_keys_recovered":295,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14082048,"data_stored":126043579,"data_compressed":7765367,"data_compressed_allocated":11243520,"data_compressed_original":124193529,"omap_allocated":14003,"internal_metadata":0},"log_size":16737,"ondisk_log_size":16737,"up":64,"acting":60,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509473,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29272,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342568,"statfs":{"total":96636764160,"available":96606789632,"internally_reserved":0,"allocated":2256896,"data_stored":5541341,"data_compressed":369266,"data_compressed_allocated":585728,"data_compressed_original":4743168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37592,"kb_used_data":4352,"kb_used_omap":33,"kb_used_meta":33182,"kb_avail":94334248,"statfs":{"total":96636764160,"available":96598269952,"internally_reserved":0,"allocated":4456448,"data_stored":28427265,"data_compressed":1611742,"data_compressed_allocated":2306048,"data_compressed_original":27269047,"omap_allocated":34021,"internal_metadata":33979163},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509473,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38068,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333772,"statfs":{"total":96636764160,"available":96597782528,"internally_reserved":0,"allocated":2781184,"data_stored":9937526,"data_compressed":605447,"data_compressed_allocated":905216,"data_compressed_original":8982528,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509474,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29540,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342300,"statfs":{"total":96636764160,"available":96606515200,"internally_reserved":0,"allocated":2568192,"data_stored":10000399,"data_compressed":603003,"data_compressed_allocated":843776,"data_compressed_original":9151005,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542176,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58272,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94313568,"statfs":{"total":96636764160,"available":96577093632,"internally_reserved":0,"allocated":4984832,"data_stored":35017940,"data_compressed":1957077,"data_compressed_allocated":2772992,"data_compressed_original":33812137,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542177,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41560,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330280,"statfs":{"total":96636764160,"available":96594206720,"internally_reserved":0,"allocated":2097152,"data_stored":6223040,"data_compressed":242651,"data_compressed_allocated":458752,"data_compressed_original":5435200,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430228,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35740,"kb_used_data":3588,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94336100,"statfs":{"total":96636764160,"available":96600166400,"internally_reserved":0,"allocated":3674112,"data_stored":18999762,"data_compressed":1116666,"data_compressed_allocated":1748992,"data_compressed_original":18051072,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56996,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3465216,"data_stored":17690677,"data_compressed":975719,"data_compressed_allocated":1466368,"data_compressed_original":16655164,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1507328,"data_stored":13191768,"data_compressed":1017405,"data_compressed_allocated":1249280,"data_compressed_original":13025821,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":9191865,"data_compressed":605447,"data_compressed_allocated":905216,"data_compressed_original":8982528,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":5490038,"data_compressed":242651,"data_compressed_allocated":458752,"data_compressed_original":5435200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3436544,"data_stored":34278799,"data_compressed":1957077,"data_compressed_allocated":2772992,"data_compressed_original":33812137,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2920448,"data_stored":27694263,"data_compressed":1611742,"data_compressed_allocated":2306048,"data_compressed_original":27269047,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1937408,"data_stored":16964069,"data_compressed":975719,"data_compressed_allocated":1466368,"data_compressed_original":16655164,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":696320,"data_stored":4795699,"data_compressed":369266,"data_compressed_allocated":585728,"data_compressed_original":4743168,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1806336,"data_stored":14437078,"data_compressed":986060,"data_compressed_allocated":1499136,"data_compressed_original":14270464,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:06.155 INFO:tasks.thrashosds.thrasher:Setting 3.1d to [0, 6, 5, 2] 2024-09-04T21:56:06.155 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.1d', '0', '6', '5', '2'] 2024-09-04T21:56:06.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.1d 0 6 5 2 2024-09-04T21:56:06.176 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:06.176 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.191 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:56:06.240 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:06.234+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:06.339 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:06.334+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:06.356 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:06.356 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.373 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:56:06.412 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-04T21:56:06.458 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:06.454+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:06.503 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:06.503 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.517 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:56:06.541 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:06.534+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:06.639 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:06.639 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.641 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:06.634+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:06.653 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:56:06.742 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:06.738+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:06.771 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:06.771 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.786 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:56:06.842 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:06.835+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:06.903 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:06.903 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:06.917 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:56:06.959 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:06.959+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:07.044 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:07.042+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:07.084 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:07.084 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:07.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:56:07.147 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:07.143+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:07.221 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:07.221 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:07.235 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:56:07.279 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:07.279+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:07.348 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:07.343+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:07.404 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:07.404 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:07.416 INFO:teuthology.orchestra.run.smithi112.stderr:set 3.1d pg_upmap_items mapping to [0->6,5->2] 2024-09-04T21:56:07.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:56:07.423 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is set 2024-09-04T21:56:07.465 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:07.459+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:07.498 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:56:07.549 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:07.542+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:07.594 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:07.595 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:07.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:56:07.644 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:07.644 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:07.650 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:07.646+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:07.659 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:07.660 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:07.675 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:07.675 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:07.691 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:07.691 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:07.725 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:07.725 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:07.734 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:07.734 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:07.735 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:07.735 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:07.745 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:07.745 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:07.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:56:07.751 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:07.751+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:07.767 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:07.767 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:07.788 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:07.788 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:07.852 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:07.846+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:07.878 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:07.879 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:07.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:56:07.953 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:07.951+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:08.054 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:08.050+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.38268431500001, 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:56:08.061 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:56:08.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:56:08.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382655382, 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.066 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:08.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:56:08.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382631735, 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:56:08.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:56:08.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.075 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382611954, 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:56:08.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:56:08.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.080 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382592559, 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:56:08.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:56:08.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382573286, 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.085 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:56:08.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382554064, 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.090 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:56:08.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.38253522700001, 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.095 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:56:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:08.099 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382515824, 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:56:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:56:08.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:08.104 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382494768, 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.38246259, 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:56:08.109 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:56:08.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382436369, 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.114 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.118 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.38240816, 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:56:08.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.38238812, 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:56:08.123 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:56:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382368738, 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.38234515000001, 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:56:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382324833, 2024-09-04T21:56:08.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:08.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:56:08.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382305281, 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.142 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:56:08.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.146 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.382285522, 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:56:08.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146632 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.6108989070000002, 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.93825028499999996, 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.151 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146632 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447333+0000", 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.66e-06 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447339+0000", 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0480000000000004e-06 2024-09-04T21:56:08.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447354+0000", 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4945e-05 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447363+0000", 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0329999999999997e-06 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447449+0000", 2024-09-04T21:56:08.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6327000000000002e-05 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447460+0000", 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1173e-05 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:04.385471+0000", 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.93801108700000002 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:04.385581+0000", 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.156 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:08.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:56:08.158 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:08.150+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:08.258 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:08.255+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:08.323 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:08.323 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:08.323 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:08.324 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:08.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:56:08.382 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:08.378+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:08.459 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:08.455+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:08.466 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:08.466 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:08.467 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:08.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:56:08.574 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:08.570+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:08.660 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:08.660 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:08.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.805133152, 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.661 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:56:08.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:56:08.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.67908588900001, 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.667 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:08.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:56:08.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.672 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.431878282, 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.673 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:56:08.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:56:08.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:56:08.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.404120492, 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.679 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:08.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:56:08.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.40400493, 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:56:08.685 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:56:08.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:56:08.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.944697593, 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:56:08.691 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:08.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:56:08.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.94447153500001, 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:56:08.697 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:56:08.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.237870356, 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.703 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:56:08.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:56:08.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 120.93956686, 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.709 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:08.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:56:08.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:56:08.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 120.759827481, 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:56:08.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:56:08.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 120.093500576, 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.721 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:56:08.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:56:08.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:145127 3.19 3.1e5da059 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.198354596000001, 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90329116399999998, 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.727 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 145127 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458503+0000", 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0029999999999999e-06 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458512+0000", 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3590000000000008e-06 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458530+0000", 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8790000000000001e-05 2024-09-04T21:56:08.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458539+0000", 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2290000000000003e-06 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458630+0000", 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0841000000000007e-05 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:08.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458644+0000", 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361754+0000", 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90310956399999998 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361791+0000", 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7728999999999997e-05 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4160 3.8 e122/121 3:1505c1f6:::rbd_data.1188b450424.0000000000000003:14 v 122'534, mlcod=0'0)", 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.164674524, 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0042635119999999997, 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:08.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492182+0000", 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492200+0000", 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8009e-05 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492229+0000", 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8469000000000001e-05 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492249+0000", 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9884000000000001e-05 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492350+0000", 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000101572 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492378+0000", 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8138999999999999e-05 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496377+0000", 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0039985480000000002 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496444+0000", 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6828000000000005e-05 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4161 3.8 e122/121 3:13882358:::rbd_data.1188b450424.0000000000000004:14 v 122'536, mlcod=0'0)", 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.161890471, 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057478820000000002, 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494967+0000", 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494982+0000", 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5226e-05 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495010+0000", 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8143000000000001e-05 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495029+0000", 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8872000000000002e-05 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495238+0000", 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020955799999999999 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495292+0000", 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3548999999999998e-05 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500587+0000", 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052953439999999996 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500712+0000", 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012510000000000001 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4171 3.8 e123/121 3:16896659:::rbd_data.11882296a3bb.00000000000001d8:12 v 123'538, mlcod=122'534)", 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.293757677, 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053660619999999996, 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363098+0000", 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363103+0000", 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9269999999999996e-06 2024-09-04T21:56:08.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363114+0000", 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1077000000000001e-05 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363137+0000", 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363420+0000", 2024-09-04T21:56:08.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028287199999999998 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363445+0000", 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5471999999999999e-05 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368381+0000", 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0049356000000000001 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368463+0000", 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2495000000000002e-05 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:276 3.1a e123/120 3:58e50b11:::rbd_data.11882296a3bb.000000000000004e:12 v 123'923, mlcod=116'919)", 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.291971612999999, 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059945689999999999, 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.745 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364884+0000", 2024-09-04T21:56:08.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9599999999999999e-07 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364893+0000", 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6000000000000007e-06 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364919+0000", 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6156000000000002e-05 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364936+0000", 2024-09-04T21:56:08.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6920000000000001e-05 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365020+0000", 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4146000000000004e-05 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365036+0000", 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6277e-05 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370836+0000", 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058004290000000002 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370878+0000", 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1344999999999998e-05 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146120 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.010467997999999, 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040719019999999996, 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146120 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646390+0000", 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0630000000000002e-06 2024-09-04T21:56:08.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646410+0000", 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9749999999999999e-05 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646453+0000", 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2879000000000001e-05 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646476+0000", 2024-09-04T21:56:08.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3886000000000001e-05 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646633+0000", 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015649199999999999 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646831+0000", 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019817899999999999 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646996+0000", 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000164745 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.648404+0000", 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001408149 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650116+0000", 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001711775 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650329+0000", 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021338100000000001 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650459+0000", 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129603 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146127 3.0 3:04d53328:::rbd_data.1188e6876e06.000000000000000e:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:08.755 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.001605639999999, 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038955439999999999, 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146127 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655249+0000", 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655252+0000", 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7269999999999999e-06 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655263+0000", 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0893000000000001e-05 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655294+0000", 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1479000000000001e-05 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655310+0000", 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5625999999999999e-05 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655473+0000", 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016301900000000001 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.655790+0000", 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000317492 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.656104+0000", 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031342799999999998 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.657718+0000", 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001614571 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.658939+0000", 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012200869999999999 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.659027+0000", 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8450999999999997e-05 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.659145+0000", 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011777099999999999 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.761 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146276 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:head [delete] snapc e=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.9632703889999998, 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051386740000000002, 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:08.762 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146276 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693587+0000", 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5600000000000001e-06 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693595+0000", 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2640000000000008e-06 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693618+0000", 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3051999999999999e-05 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693630+0000", 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2074e-05 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693705+0000", 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4867e-05 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693851+0000", 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014598700000000001 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693977+0000", 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012523100000000001 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.695244+0000", 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001267518 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698604+0000", 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003359608 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698671+0000", 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7810000000000003e-05 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698723+0000", 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1703000000000003e-05 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.767 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:360 3.1a e131/120 3:5dadb212:::rbd_data.118892c64562.000000000000003d:c v 131'931, mlcod=127'928)", 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.279227283, 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0063556569999999998, 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377640+0000", 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1776999999999999e-05 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377683+0000", 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2289000000000001e-05 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377705+0000", 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2245000000000001e-05 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377894+0000", 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018913 2024-09-04T21:56:08.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377951+0000", 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7318000000000002e-05 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.383885+0000", 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059336370000000003 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.383983+0000", 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8404000000000002e-05 2024-09-04T21:56:08.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:08.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:08.772 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:08.772 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:08.654+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:08.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:56:08.814 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:08.810+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:08.875 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:08.870+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:08.945 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:08.945 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:08.945 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:08.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:56:08.977 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:08.975+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:09.077 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:09.075+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:09.126 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:09.126 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:09.126 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:09.126 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:09.126 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:09.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:56:09.186 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:09.182+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131561 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 835584~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.260 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.228958351999999, 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111509741, 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131561 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027841+0000", 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2899999999999999e-06 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027847+0000", 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8289999999999996e-06 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027868+0000", 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1444999999999999e-05 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027885+0000", 2024-09-04T21:56:09.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6149000000000001e-05 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139001+0000", 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111116231 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139139+0000", 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013808700000000001 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139350+0000", 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021070999999999999 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131562 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 925696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.228890124000003, 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111852167, 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131562 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027909+0000", 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.164e-06 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027922+0000", 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2955999999999999e-05 2024-09-04T21:56:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027942+0000", 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9930999999999999e-05 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027955+0000", 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2916e-05 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139429+0000", 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111473537 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139560+0000", 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013118200000000001 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139760+0000", 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020048099999999999 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.269 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131563 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 4022272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.228819231000003, 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112167249, 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131563 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.271 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028594+0000", 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061550000000000005 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028604+0000", 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3889999999999994e-06 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028627+0000", 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3309e-05 2024-09-04T21:56:09.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028644+0000", 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7062e-05 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139847+0000", 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111203128 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139975+0000", 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012794799999999999 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140146+0000", 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017091299999999999 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131564 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1015808~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:09.274 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.228123885999999, 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11196729499999999, 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131564 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034391+0000", 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005716881 2024-09-04T21:56:09.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034402+0000", 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0448999999999999e-05 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034426+0000", 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3975e-05 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034445+0000", 2024-09-04T21:56:09.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9386e-05 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140259+0000", 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10581388899999999 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140415+0000", 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015661599999999999 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140642+0000", 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226099 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131577 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2097152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.221833478999997, 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111297805, 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.279 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131577 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034966+0000", 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034971+0000", 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0200000000000002e-06 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035005+0000", 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3763999999999997e-05 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035023+0000", 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8232000000000001e-05 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.145896+0000", 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.110872963 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146021+0000", 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124888 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146262+0000", 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024174099999999999 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131578 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2187264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.221751454, 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111709911, 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131578 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035048+0000", 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3400000000000001e-06 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035054+0000", 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7660000000000004e-06 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035072+0000", 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8224999999999999e-05 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035085+0000", 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3152999999999999e-05 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146367+0000", 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111282279 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146521+0000", 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015351399999999999 2024-09-04T21:56:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146757+0000", 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000235634 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131579 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2277376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.221694585000002, 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11209353800000001, 2024-09-04T21:56:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131579 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035105+0000", 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2470000000000001e-06 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035110+0000", 2024-09-04T21:56:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.237e-06 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035131+0000", 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1277999999999999e-05 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035153+0000", 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1959000000000001e-05 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146846+0000", 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11169317600000001 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146984+0000", 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137095 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147197+0000", 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000213546 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131580 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2367488~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.221625478999997, 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112535119, 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131580 2024-09-04T21:56:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042368+0000", 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071953859999999998 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042379+0000", 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1069e-05 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042403+0000", 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4158000000000001e-05 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042422+0000", 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8983000000000001e-05 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147304+0000", 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10488141500000001 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147478+0000", 2024-09-04T21:56:09.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386300000000001 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147708+0000", 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023024499999999999 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131596 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 49152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.213728781999997, 2024-09-04T21:56:09.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111536857, 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131596 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043071+0000", 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3799999999999999e-06 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043077+0000", 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043096+0000", 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9681e-05 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043113+0000", 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6416e-05 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154171+0000", 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111058771 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154353+0000", 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018107400000000001 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154606+0000", 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025366800000000002 2024-09-04T21:56:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131597 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3809280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.213665325999997, 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111867072, 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131597 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043134+0000", 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2529999999999999e-06 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043147+0000", 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2693e-05 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043164+0000", 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043177+0000", 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729e-05 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154692+0000", 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111515138 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154825+0000", 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013267800000000001 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155000+0000", 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017526900000000001 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131598 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.21360146, 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112278195, 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131598 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050331+0000", 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071345200000000001 2024-09-04T21:56:09.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050342+0000", 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1189000000000001e-05 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050367+0000", 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4621e-05 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050387+0000", 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819000000000001e-05 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155115+0000", 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104728349 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155274+0000", 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015923399999999999 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155475+0000", 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020046300000000001 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131614 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 589824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.205634957000001, 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111567737, 2024-09-04T21:56:09.311 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131614 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051164+0000", 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2160000000000001e-06 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051170+0000", 2024-09-04T21:56:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2390000000000002e-06 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051187+0000", 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7649e-05 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051200+0000", 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2993000000000001e-05 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162333+0000", 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111132965 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162500+0000", 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016687900000000001 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162731+0000", 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023079599999999999 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131615 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.205579305000001, 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11193702, 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.316 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131615 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051220+0000", 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051225+0000", 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1399999999999999e-06 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051242+0000", 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6988999999999999e-05 2024-09-04T21:56:09.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051255+0000", 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3261999999999999e-05 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162787+0000", 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111532033 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162923+0000", 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013530399999999999 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163156+0000", 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233095 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131616 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.320 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.205524734999997, 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112285617, 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131616 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058214+0000", 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069405279999999996 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058226+0000", 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1732999999999999e-05 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058253+0000", 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7574999999999999e-05 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058276+0000", 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2334999999999999e-05 2024-09-04T21:56:09.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163207+0000", 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104930906 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163336+0000", 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129639 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163559+0000", 2024-09-04T21:56:09.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022290100000000001 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131631 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1220608~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.197896227000001, 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111259443, 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.325 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131631 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058903+0000", 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.133e-06 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058909+0000", 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6890000000000001e-06 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058926+0000", 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7166000000000001e-05 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058943+0000", 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7028999999999999e-05 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169875+0000", 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11093157400000001 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169996+0000", 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012140499999999999 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170161+0000", 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016544699999999999 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131632 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1613824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.197833297999999, 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111614907, 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131632 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058966+0000", 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058974+0000", 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9559999999999997e-06 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059013+0000", 2024-09-04T21:56:09.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8688e-05 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059027+0000", 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170250+0000", 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111222396 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170381+0000", 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013136800000000001 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170580+0000", 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019870700000000001 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131633 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1703936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.334 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.197746283999997, 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11190109299999999, 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131633 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059053+0000", 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.353e-06 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059067+0000", 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3482999999999999e-05 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059104+0000", 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7657999999999998e-05 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059129+0000", 2024-09-04T21:56:09.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4380999999999999e-05 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170639+0000", 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111510427 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170761+0000", 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122302 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170953+0000", 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019148900000000001 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131634 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.197646364000001, 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11221152299999999, 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.339 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131634 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059154+0000", 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059160+0000", 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:56:09.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059199+0000", 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9190999999999998e-05 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059220+0000", 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0499000000000001e-05 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171022+0000", 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11180261599999999 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171158+0000", 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135401 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171363+0000", 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020553999999999999 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131635 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1794048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.197556902999999, 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112472239, 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.344 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131635 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066220+0000", 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069786229999999998 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066229+0000", 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066251+0000", 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1829000000000001e-05 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066273+0000", 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2722e-05 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171426+0000", 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10515273 2024-09-04T21:56:09.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171548+0000", 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012224 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171714+0000", 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000165289 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131652 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2785280~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.189666615, 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111334216, 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131652 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074601+0000", 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074699279999999998 2024-09-04T21:56:09.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074617+0000", 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5882000000000001e-05 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074651+0000", 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4100999999999998e-05 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074674+0000", 2024-09-04T21:56:09.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2858999999999999e-05 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178090+0000", 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10341562999999999 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178247+0000", 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156673 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178466+0000", 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021914300000000001 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.353 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:09.354 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:56:09.355 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:09.347+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:09.456 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:09.451+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:09.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-04T21:56:09.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:09.536 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:09.536 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:09.536 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:09.554 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:56:09.556 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:09.551+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:09.657 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:09.650+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:09.682 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:09.683 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:09.683 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:09.683 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:09.683 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:09.698 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:56:09.758 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:09.755+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:09.805 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:56:09.857 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:09.857 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:09.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:09.857 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 99.352425819999993, 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:56:09.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:56:09.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007649434000001, 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:56:09.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:56:09.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007620829000004, 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.867 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:56:09.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007600792000005, 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:56:09.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:56:09.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007581569999999, 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:56:09.876 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:56:09.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:56:09.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007562778999997, 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:56:09.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:56:09.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007530837000004, 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.885 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.889 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007502307999999, 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:56:09.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007482697, 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.894 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.898 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007464404000004, 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:56:09.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:56:09.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007441589999999, 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:56:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.907 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007422453000004, 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:56:09.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007401787000006, 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.912 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:56:09.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.916 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.00737101, 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:56:09.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:56:09.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007351642000003, 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.921 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:56:09.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007327683, 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.926 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:56:09.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.930 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007280433999995, 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007254755999995, 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.935 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.939 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.00722073, 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:56:09.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.007196328999996, 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:09.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:09.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:09.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:09.948 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:09.950 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:56:09.950 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:09.951 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:09.951 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:09.947+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:09.963 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:09.963 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:09.979 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:09.979 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:09.996 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:09.996 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:10.016 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:10.016 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:10.036 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:10.051 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:10.047+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:10.057 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:10.058 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:10.077 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:10.077 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:10.084 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:10.085 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:10.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:56:10.187 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:10.187+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:10.252 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:10.251+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:10.261 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:10.261 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:10.276 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:56:10.353 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:10.346+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:10.361 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is set 2024-09-04T21:56:10.453 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:10.454 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:10.454 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-04T21:56:10.450+0000 7fea34e66640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26415) UID: 0 2024-09-04T21:56:10.469 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:56:10.554 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:10.551+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:10.591 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:10.591 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:10.606 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:56:10.699 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:10.699+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:10.756 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:10.755+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:10.776 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:10.777 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:10.791 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:56:10.880 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:10.879+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:10.958 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:10.955+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:10.959 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:10.959 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:10.974 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:56:11.059 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:11.059+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:11.142 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:11.142 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:11.156 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:56:11.199 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:11.199+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:11.260 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:11.259+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:11.320 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:11.320 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:11.336 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:56:11.382 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:11.379+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:11.462 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:11.459+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:11.500 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:11.500 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:11.515 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:56:11.562 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:11.558+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:11.664 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:11.658+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:11.688 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:11.689 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:11.703 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:56:11.764 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:11.763+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:11.825 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:11.826 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:11.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:56:11.883 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:11.883+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:11.965 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:11.958+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:12.007 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:12.008 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:12.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:56:12.066 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:12.063+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:12.094 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-04T21:56:12.156 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:12.157 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:12.167 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:12.162+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:12.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:56:12.244 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:12.244 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:12.263 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:12.263 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:12.267 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:12.262+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:12.280 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:12.280 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:12.296 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:12.297 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:12.317 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:12.317 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:12.330 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:12.331 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:12.331 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:12.338 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:12.338 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:12.345 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:56:12.359 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:12.359 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:12.380 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:12.380 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:12.380 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:12.374+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:12.389 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:56:12.460 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-04T21:56:12.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-04T21:56:12.460 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-04T21:56:12.460 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-04T21:56:12.484 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6453 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 448512~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807803233, 2024-09-04T21:56:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039516146000000002, 2024-09-04T21:56:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6453 2024-09-04T21:56:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675545+0000", 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675546+0000", 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1199999999999998e-07 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675548+0000", 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819999999999998e-06 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675554+0000", 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8320000000000002e-06 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675566+0000", 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2455e-05 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714846+0000", 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039279385 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714889+0000", 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3501000000000002e-05 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715062+0000", 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017257899999999999 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.492 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6454 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 452608~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80777430000001, 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039747855999999998, 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6454 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.493 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675574+0000", 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675575+0000", 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3099999999999998e-07 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675577+0000", 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3999999999999999e-06 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675585+0000", 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1570000000000003e-06 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675590+0000", 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3079999999999998e-06 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715079+0000", 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039489303000000003 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715121+0000", 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1822e-05 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715322+0000", 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000201335 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6455 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 456704~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807750653, 2024-09-04T21:56:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039986414999999997, 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6455 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675598+0000", 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675600+0000", 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9889999999999999e-06 2024-09-04T21:56:12.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675607+0000", 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2960000000000004e-06 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675611+0000", 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.651e-06 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715353+0000", 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039741242000000003 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715401+0000", 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8650999999999999e-05 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715584+0000", 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000183191 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.501 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6456 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 460800~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80773087199999, 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040242050000000001, 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.502 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6456 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675618+0000", 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.72e-07 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675620+0000", 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675626+0000", 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0909999999999998e-06 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675631+0000", 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6210000000000003e-06 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715603+0000", 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.039971948 2024-09-04T21:56:12.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715666+0000", 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3526000000000001e-05 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715860+0000", 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019364800000000001 2024-09-04T21:56:12.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.507 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.507 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.507 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6457 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 464896~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.508 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:12.508 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807711477, 2024-09-04T21:56:12.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040491483000000002, 2024-09-04T21:56:12.508 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.508 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.508 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.509 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6457 2024-09-04T21:56:12.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.510 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:12.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675637+0000", 2024-09-04T21:56:12.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675638+0000", 2024-09-04T21:56:12.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5600000000000001e-07 2024-09-04T21:56:12.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675639+0000", 2024-09-04T21:56:12.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8640000000000001e-06 2024-09-04T21:56:12.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675645+0000", 2024-09-04T21:56:12.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.032e-06 2024-09-04T21:56:12.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675650+0000", 2024-09-04T21:56:12.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5950000000000001e-06 2024-09-04T21:56:12.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715878+0000", 2024-09-04T21:56:12.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040228084999999997 2024-09-04T21:56:12.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715951+0000", 2024-09-04T21:56:12.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3090999999999995e-05 2024-09-04T21:56:12.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716129+0000", 2024-09-04T21:56:12.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017746 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6458 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 468992~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80769220400001, 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040823195999999999, 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.518 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6458 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675657+0000", 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675659+0000", 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9039999999999999e-06 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675665+0000", 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9560000000000002e-06 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675669+0000", 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5900000000000001e-06 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716147+0000", 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040477807999999997 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716215+0000", 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8131000000000002e-05 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716480+0000", 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026444700000000001 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6459 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 473088~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807672982, 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041124198000000001, 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6459 2024-09-04T21:56:12.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675676+0000", 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5999999999999999e-07 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675678+0000", 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.832e-06 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675684+0000", 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9839999999999998e-06 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675688+0000", 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5719999999999996e-06 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716499+0000", 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.040810875000000003 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716544+0000", 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5053999999999999e-05 2024-09-04T21:56:12.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716800+0000", 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025552099999999998 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6460 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 477184~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807654145, 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041408584999999998, 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.527 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6460 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675695+0000", 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5400000000000002e-07 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675697+0000", 2024-09-04T21:56:12.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.753e-06 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675702+0000", 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6289999999999998e-06 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675707+0000", 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5920000000000002e-06 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716829+0000", 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041122009000000001 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716875+0000", 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5812000000000001e-05 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717103+0000", 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022843599999999999 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6461 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 481280~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807634742, 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041784910000000001, 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6461 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.532 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675714+0000", 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.32e-07 2024-09-04T21:56:12.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675717+0000", 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1560000000000002e-06 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675723+0000", 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6610000000000003e-06 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675728+0000", 2024-09-04T21:56:12.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8060000000000002e-06 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717135+0000", 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041406951999999997 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717216+0000", 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0785000000000007e-05 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717499+0000", 2024-09-04T21:56:12.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028311800000000001 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6462 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 485376~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807613686, 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042078658999999997, 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.536 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6462 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675735+0000", 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3700000000000001e-07 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675741+0000", 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4960000000000004e-06 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675748+0000", 2024-09-04T21:56:12.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2509999999999997e-06 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675760+0000", 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1372999999999999e-05 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717534+0000", 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.041774138000000002 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717587+0000", 2024-09-04T21:56:12.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3304999999999999e-05 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717814+0000", 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022665900000000001 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6463 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 489472~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807581508, 2024-09-04T21:56:12.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042358595999999998, 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6463 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675767+0000", 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675768+0000", 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2399999999999999e-07 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675775+0000", 2024-09-04T21:56:12.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0119999999999999e-06 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675781+0000", 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7179999999999999e-06 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675786+0000", 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.848e-06 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717851+0000", 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.042064613000000001 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717901+0000", 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0117000000000001e-05 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718126+0000", 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022486399999999999 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6464 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 493568~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807555287, 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044191596, 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.545 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6464 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675793+0000", 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675794+0000", 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2800000000000002e-07 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675796+0000", 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0880000000000002e-06 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675802+0000", 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.251e-06 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675812+0000", 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0071000000000001e-05 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718154+0000", 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.04234156 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.718226+0000", 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1781999999999998e-05 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719985+0000", 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001759416 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6465 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 497664~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80752707800001, 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044457139, 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.549 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6465 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:12.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675822+0000", 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675824+0000", 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1239999999999999e-06 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675830+0000", 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3559999999999998e-06 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675835+0000", 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7260000000000002e-06 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719823+0000", 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.043987330999999998 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.719917+0000", 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4919000000000004e-05 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720279+0000", 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000361314 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6466 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 501760~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.553 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807507038, 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044949898000000002, 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6466 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:12.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675842+0000", 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675844+0000", 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675850+0000", 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.66e-06 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675854+0000", 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5349999999999998e-06 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720311+0000", 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044456477000000001 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720402+0000", 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1893999999999998e-05 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720792+0000", 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038908600000000002 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6467 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 505856~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80748765600001, 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045743728999999997, 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6467 2024-09-04T21:56:12.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675861+0000", 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675862+0000", 2024-09-04T21:56:12.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5299999999999999e-07 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675864+0000", 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1270000000000001e-06 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675870+0000", 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3280000000000002e-06 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675875+0000", 2024-09-04T21:56:12.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.271e-06 2024-09-04T21:56:12.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720823+0000", 2024-09-04T21:56:12.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044947272000000003 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.720916+0000", 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3391000000000001e-05 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721605+0000", 2024-09-04T21:56:12.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068888700000000001 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6468 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 509952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807464068, 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046564292, 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.565 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6468 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675885+0000", 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9299999999999999e-07 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675887+0000", 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0619999999999999e-06 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675894+0000", 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4749999999999998e-06 2024-09-04T21:56:12.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675898+0000", 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7559999999999999e-06 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721637+0000", 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045738544999999999 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.721724+0000", 2024-09-04T21:56:12.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7590999999999995e-05 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722449+0000", 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00072446999999999995 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6469 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 514048~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80744375099999, 2024-09-04T21:56:12.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047021614000000003, 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6469 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675905+0000", 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3000000000000002e-07 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675907+0000", 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8899999999999999e-06 2024-09-04T21:56:12.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675913+0000", 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1430000000000003e-06 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675918+0000", 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6700000000000002e-06 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722485+0000", 2024-09-04T21:56:12.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.046566952000000002 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722581+0000", 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5853999999999994e-05 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722927+0000", 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034577499999999999 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6470 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 518144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.807424199, 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047485904000000002, 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6470 2024-09-04T21:56:12.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675925+0000", 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.39e-07 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675927+0000", 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.844e-06 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675933+0000", 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3450000000000002e-06 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675938+0000", 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6569999999999996e-06 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.722962+0000", 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047024020999999999 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723054+0000", 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2159999999999999e-05 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723410+0000", 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00035653799999999999 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:6471 3.1b 3:da83c0d6:::rbd_data.1188e7c968d9.0000000000000005:4 [read 522240~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.80740444, 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047943240999999998, 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.578 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 6471 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675944+0000", 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675945+0000", 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3599999999999999e-07 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675950+0000", 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9470000000000002e-06 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675956+0000", 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0719999999999996e-06 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.675968+0000", 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2532e-05 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723442+0000", 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.047473619000000002 2024-09-04T21:56:12.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723535+0000", 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3499999999999996e-05 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.723888+0000", 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000352235 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146632 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.0360178250000001, 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.93825028499999996, 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146632 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447331+0000", 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447333+0000", 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.66e-06 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447339+0000", 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0480000000000004e-06 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447354+0000", 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4945e-05 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447363+0000", 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0329999999999997e-06 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447449+0000", 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6327000000000002e-05 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.447460+0000", 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1173e-05 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:04.385471+0000", 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.93801108700000002 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:04.385581+0000", 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:12.587 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:12.587 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:12.478+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:12.588 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:56:12.590 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:12.587+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:12.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-04T21:56:12.690 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:12.687+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:12.746 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:12.746 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:12.746 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:12.746 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:12.761 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:56:12.792 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:12.791+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:12.849 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-04T21:56:12.877 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:12.877 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:12.877 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:12.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:56:12.893 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:12.891+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:12.994 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:12.990+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:13.005 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-04T21:56:13.050 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:13.050 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1330 3.19 3:9f144cd4:::rbd_data.1188e7c968d9.0000000000000090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 130.19718362500001, 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0047364740000000001, 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1330 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851722+0000", 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851724+0000", 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5610000000000001e-06 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851732+0000", 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4359999999999996e-06 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851754+0000", 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2254000000000001e-05 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851767+0000", 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3097e-05 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.851941+0000", 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386999999999999 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852148+0000", 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020734400000000001 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.852432+0000", 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028378499999999999 2024-09-04T21:56:13.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.854215+0000", 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017825740000000001 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856339+0000", 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0021248259999999998 2024-09-04T21:56:13.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856403+0000", 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3944999999999998e-05 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.856458+0000", 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4781999999999998e-05 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.057 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1421 3.0 3:0677d33a:::rbd_data.1188e7c968d9.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 501295] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 130.071136362, 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005041869, 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1421 2024-09-04T21:56:13.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977769+0000", 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.977771+0000", 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.934e-06 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978134+0000", 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036306100000000003 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978152+0000", 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7621999999999998e-05 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978160+0000", 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9440000000000005e-06 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978310+0000", 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015019600000000001 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.978719+0000", 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040954300000000001 2024-09-04T21:56:13.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.979389+0000", 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066977400000000004 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982479+0000", 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003090237 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982525+0000", 2024-09-04T21:56:13.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6003e-05 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982685+0000", 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000159896 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:02.982811+0000", 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125659 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1555 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [set-alloc-hint object_size 524288 write_size 524288,write 117980~13092] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.823928755, 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040172660000000002, 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1555 2024-09-04T21:56:13.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224977+0000", 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.224980+0000", 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8500000000000004e-06 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225013+0000", 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2447000000000003e-05 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225052+0000", 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9146000000000001e-05 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225070+0000", 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8063e-05 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225202+0000", 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000131979 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225456+0000", 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000253711 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.225706+0000", 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025049499999999999 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228163+0000", 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002456869 2024-09-04T21:56:13.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228826+0000", 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00066299100000000004 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228918+0000", 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1586e-05 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.228994+0000", 2024-09-04T21:56:13.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6129000000000002e-05 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1560 3.4 3:27fdc49b:::rbd_data.1188e7c968d9.00000000000000e0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.79617096499999, 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.004184124, 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.070 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1560 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252734+0000", 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252738+0000", 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2399999999999999e-06 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252750+0000", 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.204e-05 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252787+0000", 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.735e-05 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252805+0000", 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088999999999999e-05 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252984+0000", 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017919499999999999 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253302+0000", 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031748099999999999 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.253621+0000", 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031939799999999998 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.255882+0000", 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022607999999999999 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256732+0000", 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084970300000000005 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256842+0000", 2024-09-04T21:56:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110449 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256919+0000", 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6378999999999995e-05 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:1563 3.4 3:27c98779:::rbd_data.1188e7c968d9.000000000000017a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.796055403, 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046151910000000003, 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1563 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252850+0000", 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252851+0000", 2024-09-04T21:56:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.023e-06 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252855+0000", 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.464e-06 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252870+0000", 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4966e-05 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.252879+0000", 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-06 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254464+0000", 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001584787 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254696+0000", 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023228800000000001 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.254991+0000", 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000294324 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.256586+0000", 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001595223 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257331+0000", 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074550699999999996 2024-09-04T21:56:13.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257408+0000", 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6896000000000004e-05 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:03.257465+0000", 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6913000000000002e-05 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7156 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 126.336748066, 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041987429999999996, 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7156 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.083 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712157+0000", 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712160+0000", 2024-09-04T21:56:13.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6860000000000002e-06 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712315+0000", 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015454800000000001 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712340+0000", 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5516000000000001e-05 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712353+0000", 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2847000000000001e-05 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712466+0000", 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000113587 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712666+0000", 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000199567 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712934+0000", 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026810899999999999 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715927+0000", 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029926079999999999 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716228+0000", 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030077899999999999 2024-09-04T21:56:13.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716304+0000", 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6673000000000003e-05 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716356+0000", 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1823000000000002e-05 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7157 3.4 3:245a88e6:::rbd_data.118867f9efa3.000000000000000e:head [write 0~0] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 126.336522008, 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046360250000000002, 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7157 2024-09-04T21:56:13.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712383+0000", 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712384+0000", 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712388+0000", 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7400000000000002e-06 2024-09-04T21:56:13.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712408+0000", 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9783999999999999e-05 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.712422+0000", 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3986e-05 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714003+0000", 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015810799999999999 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714103+0000", 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9774999999999999e-05 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.714371+0000", 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026812800000000002 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.715933+0000", 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0015622699999999999 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716912+0000", 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00097883400000000004 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.716971+0000", 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9358000000000003e-05 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:06.717019+0000", 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8189999999999998e-05 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7497 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write 1816560~18448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.629920829, 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0065621619999999999, 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.096 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7497 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418985+0000", 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.418989+0000", 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0439999999999997e-06 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419045+0000", 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6870000000000003e-05 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419082+0000", 2024-09-04T21:56:13.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6881999999999997e-05 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419111+0000", 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9e-05 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.419551+0000", 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000439558 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420161+0000", 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061050299999999998 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.420581+0000", 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000419317 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.423451+0000", 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028703219999999998 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425317+0000", 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018660510000000001 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425419+0000", 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010238600000000001 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.425547+0000", 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000127229 2024-09-04T21:56:13.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7610 3.19 3:9d01c322:::rbd_data.1188b1294ada.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.331617333, 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040433860000000004, 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.102 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7610 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717288+0000", 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717289+0000", 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729999999999999e-06 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717296+0000", 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5030000000000002e-06 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717344+0000", 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8270999999999999e-05 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.717361+0000", 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6694999999999999e-05 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718523+0000", 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011618710000000001 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.718918+0000", 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039508700000000002 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.719179+0000", 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026108099999999998 2024-09-04T21:56:13.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.720590+0000", 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014112269999999999 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721147+0000", 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055681800000000005 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721223+0000", 2024-09-04T21:56:13.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6304999999999995e-05 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.721331+0000", 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000108255 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7729 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1835008~39369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.151877954, 2024-09-04T21:56:13.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051018469999999996, 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7729 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:13.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897027+0000", 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897032+0000", 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.967e-06 2024-09-04T21:56:13.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897091+0000", 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8773000000000001e-05 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897188+0000", 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6682999999999996e-05 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897235+0000", 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6903999999999997e-05 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897407+0000", 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017221099999999999 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.897794+0000", 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038753399999999998 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.898235+0000", 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044090200000000001 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.900542+0000", 2024-09-04T21:56:13.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023066079999999999 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.901943+0000", 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0014010649999999999 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902030+0000", 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7393999999999997e-05 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:07.902129+0000", 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8806000000000002e-05 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:7862 3.4 3:222e91ff:::rbd_data.1188b1294ada.000000000000003d:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.48555104899999, 2024-09-04T21:56:13.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0064071140000000002, 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 7862 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563354+0000", 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563358+0000", 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1459999999999998e-06 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563373+0000", 2024-09-04T21:56:13.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4365000000000001e-05 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563427+0000", 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3990999999999999e-05 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563451+0000", 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3964e-05 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563631+0000", 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000180094 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.563857+0000", 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226586 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.564455+0000", 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00059715199999999999 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.568522+0000", 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040675529999999998 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569602+0000", 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010796709999999999 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569692+0000", 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9889999999999995e-05 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:08.569761+0000", 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9702000000000005e-05 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:145127 3.19 3.1e5da059 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.590405068999999, 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90329116399999998, 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.122 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 145127 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458500+0000", 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458503+0000", 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0029999999999999e-06 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458512+0000", 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3590000000000008e-06 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458530+0000", 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8790000000000001e-05 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458539+0000", 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2290000000000003e-06 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458630+0000", 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0841000000000007e-05 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:49.458644+0000", 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361754+0000", 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.90310956399999998 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:50.361791+0000", 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7728999999999997e-05 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4160 3.8 e122/121 3:1505c1f6:::rbd_data.1188b450424.0000000000000003:14 v 122'534, mlcod=0'0)", 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.556724997, 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0042635119999999997, 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.127 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492180+0000", 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492182+0000", 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.063e-06 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492200+0000", 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8009e-05 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492229+0000", 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8469000000000001e-05 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492249+0000", 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9884000000000001e-05 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492350+0000", 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000101572 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.492378+0000", 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8138999999999999e-05 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496377+0000", 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0039985480000000002 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.496444+0000", 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6828000000000005e-05 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.131 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4161 3.8 e122/121 3:13882358:::rbd_data.1188b450424.0000000000000004:14 v 122'536, mlcod=0'0)", 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.553940944000001, 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0057478820000000002, 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494964+0000", 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494967+0000", 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0899999999999999e-06 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.494982+0000", 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5226e-05 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495010+0000", 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8143000000000001e-05 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495029+0000", 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8872000000000002e-05 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495238+0000", 2024-09-04T21:56:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020955799999999999 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.495292+0000", 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3548999999999998e-05 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500587+0000", 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052953439999999996 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:54.500712+0000", 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012510000000000001 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.3.0:4171 3.8 e123/121 3:16896659:::rbd_data.11882296a3bb.00000000000001d8:12 v 123'538, mlcod=122'534)", 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.68580815, 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0053660619999999996, 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363097+0000", 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363098+0000", 2024-09-04T21:56:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363103+0000", 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9269999999999996e-06 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363114+0000", 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1077000000000001e-05 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363137+0000", 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363420+0000", 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028287199999999998 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.363445+0000", 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5471999999999999e-05 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368381+0000", 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0049356000000000001 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.368463+0000", 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2495000000000002e-05 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:276 3.1a e123/120 3:58e50b11:::rbd_data.11882296a3bb.000000000000004e:12 v 123'923, mlcod=116'919)", 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.684022085999999, 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059945689999999999, 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364883+0000", 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364884+0000", 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9599999999999999e-07 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364893+0000", 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6000000000000007e-06 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364919+0000", 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6156000000000002e-05 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.364936+0000", 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6920000000000001e-05 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365020+0000", 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4146000000000004e-05 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.365036+0000", 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6277e-05 2024-09-04T21:56:13.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370836+0000", 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058004290000000002 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:55.370878+0000", 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1344999999999998e-05 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146120 3.0 3:0216e362:::rbd_data.1188e6876e06.0000000000000005:head [delete] snapc f=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.402518471, 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040719019999999996, 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146120 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646387+0000", 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646390+0000", 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0630000000000002e-06 2024-09-04T21:56:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646410+0000", 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9749999999999999e-05 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646453+0000", 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2879000000000001e-05 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646476+0000", 2024-09-04T21:56:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3886000000000001e-05 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646633+0000", 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015649199999999999 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646831+0000", 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019817899999999999 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 2", 2024-09-04T21:56:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.646996+0000", 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000164745 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.648404+0000", 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001408149 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650116+0000", 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001711775 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650329+0000", 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021338100000000001 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:58.650459+0000", 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129603 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:146276 3.19 3:9f902706:::rbd_data.118852763fbb.0000000000000000:head [delete] snapc e=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.355320861999999, 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0051386740000000002, 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 146276 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693584+0000", 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693587+0000", 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5600000000000001e-06 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693595+0000", 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2640000000000008e-06 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693618+0000", 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3051999999999999e-05 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693630+0000", 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2074e-05 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693705+0000", 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4867e-05 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693851+0000", 2024-09-04T21:56:13.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014598700000000001 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.693977+0000", 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012523100000000001 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.695244+0000", 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001267518 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698604+0000", 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003359608 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698671+0000", 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7810000000000003e-05 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:59.698723+0000", 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1703000000000003e-05 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.4.0:360 3.1a e131/120 3:5dadb212:::rbd_data.118892c64562.000000000000003d:c v 131'931, mlcod=127'928)", 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.6712777560000003, 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0063556569999999998, 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377628+0000", 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377640+0000", 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1776999999999999e-05 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377683+0000", 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2289000000000001e-05 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377705+0000", 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2245000000000001e-05 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377894+0000", 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018913 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.377951+0000", 2024-09-04T21:56:13.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7318000000000002e-05 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.383885+0000", 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059336370000000003 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:03.383983+0000", 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8404000000000002e-05 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:147144 3.19 3:9822bcc9:::rbd_object_map.118835a169af.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:56:09.585999+0000", 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.4629062660000001, 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040152829999999997, 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 147144 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.163 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.585999+0000", 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.585999+0000", 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586000+0000", 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3069999999999999e-06 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586006+0000", 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1159999999999998e-06 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586017+0000", 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1116e-05 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586024+0000", 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5630000000000003e-06 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586094+0000", 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9509000000000001e-05 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586216+0000", 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012228400000000001 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3", 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.586366+0000", 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000149777 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.587237+0000", 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00087147500000000003 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.589932+0000", 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026952249999999999 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.589966+0000", 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3772999999999997e-05 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:56:09.590014+0000", 2024-09-04T21:56:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8137999999999999e-05 2024-09-04T21:56:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.169 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:13.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:56:13.171 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:13.167+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:13.209 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-04T21:56:13.271 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:13.271+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:13.361 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:13.361 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:13.361 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:13.361 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:13.372 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:13.366+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:13.375 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:56:13.403 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-04T21:56:13.461 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:56:13.482 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:13.478+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:13.566 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:13.567 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:13.574 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:13.571+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:13.584 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:56:13.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-04T21:56:13.694 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:13.690+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:13.700 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.700 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131561 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 835584~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.670927122999998, 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111509741, 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131561 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027840+0000", 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027841+0000", 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2899999999999999e-06 2024-09-04T21:56:13.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027847+0000", 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8289999999999996e-06 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027868+0000", 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1444999999999999e-05 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027885+0000", 2024-09-04T21:56:13.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6149000000000001e-05 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139001+0000", 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111116231 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139139+0000", 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013808700000000001 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139350+0000", 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021070999999999999 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131562 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 925696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.670858895000002, 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111852167, 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.705 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131562 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027908+0000", 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027909+0000", 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.164e-06 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027922+0000", 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2955999999999999e-05 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027942+0000", 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9930999999999999e-05 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027955+0000", 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2916e-05 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139429+0000", 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111473537 2024-09-04T21:56:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139560+0000", 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013118200000000001 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139760+0000", 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020048099999999999 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131563 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 4022272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.670788002000002, 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112167249, 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131563 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.710 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.027979+0000", 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028594+0000", 2024-09-04T21:56:13.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061550000000000005 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028604+0000", 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3889999999999994e-06 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028627+0000", 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3309e-05 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028644+0000", 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7062e-05 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139847+0000", 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111203128 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.139975+0000", 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012794799999999999 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140146+0000", 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017091299999999999 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131564 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1015808~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.670092656999998, 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11196729499999999, 2024-09-04T21:56:13.714 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131564 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.028674+0000", 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034391+0000", 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005716881 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034402+0000", 2024-09-04T21:56:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0448999999999999e-05 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034426+0000", 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3975e-05 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034445+0000", 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9386e-05 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140259+0000", 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10581388899999999 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140415+0000", 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015661599999999999 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.140642+0000", 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000226099 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131577 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2097152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.663802250000003, 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111297805, 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131577 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034965+0000", 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034966+0000", 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.034971+0000", 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0200000000000002e-06 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035005+0000", 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3763999999999997e-05 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035023+0000", 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8232000000000001e-05 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.145896+0000", 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.110872963 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146021+0000", 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124888 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146262+0000", 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024174099999999999 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131578 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2187264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.663720224999999, 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111709911, 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131578 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035047+0000", 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035048+0000", 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3400000000000001e-06 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035054+0000", 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7660000000000004e-06 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035072+0000", 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8224999999999999e-05 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035085+0000", 2024-09-04T21:56:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3152999999999999e-05 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146367+0000", 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111282279 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146521+0000", 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015351399999999999 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146757+0000", 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000235634 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131579 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2277376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.663663356000001, 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11209353800000001, 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131579 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035104+0000", 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035105+0000", 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2470000000000001e-06 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035110+0000", 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.237e-06 2024-09-04T21:56:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035131+0000", 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1277999999999999e-05 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035153+0000", 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1959000000000001e-05 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146846+0000", 2024-09-04T21:56:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11169317600000001 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.146984+0000", 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137095 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147197+0000", 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000213546 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131580 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2367488~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.663594250000003, 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112535119, 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131580 2024-09-04T21:56:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.035173+0000", 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042368+0000", 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071953859999999998 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042379+0000", 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1069e-05 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042403+0000", 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4158000000000001e-05 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.042422+0000", 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8983000000000001e-05 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147304+0000", 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10488141500000001 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147478+0000", 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017386300000000001 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.147708+0000", 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023024499999999999 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131596 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 49152~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.655697553000003, 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111536857, 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131596 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043069+0000", 2024-09-04T21:56:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043071+0000", 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3799999999999999e-06 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043077+0000", 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8669999999999999e-06 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043096+0000", 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9681e-05 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043113+0000", 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6416e-05 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154171+0000", 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111058771 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154353+0000", 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018107400000000001 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154606+0000", 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025366800000000002 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131597 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3809280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.655634096999997, 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111867072, 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131597 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043133+0000", 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043134+0000", 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2529999999999999e-06 2024-09-04T21:56:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043147+0000", 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2693e-05 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043164+0000", 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7312e-05 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043177+0000", 2024-09-04T21:56:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2729e-05 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154692+0000", 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111515138 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.154825+0000", 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013267800000000001 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155000+0000", 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017526900000000001 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131598 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.655570230999999, 2024-09-04T21:56:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112278195, 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131598 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.043197+0000", 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050331+0000", 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0071345200000000001 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050342+0000", 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1189000000000001e-05 2024-09-04T21:56:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050367+0000", 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4621e-05 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.050387+0000", 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9819000000000001e-05 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155115+0000", 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104728349 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155274+0000", 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015923399999999999 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.155475+0000", 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020046300000000001 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131614 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 589824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.647603728, 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111567737, 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131614 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051163+0000", 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051164+0000", 2024-09-04T21:56:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2160000000000001e-06 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051170+0000", 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2390000000000002e-06 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051187+0000", 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7649e-05 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051200+0000", 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2993000000000001e-05 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162333+0000", 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111132965 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162500+0000", 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016687900000000001 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162731+0000", 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023079599999999999 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131615 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 712704~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.647548076, 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11193702, 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131615 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051219+0000", 2024-09-04T21:56:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051220+0000", 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051225+0000", 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1399999999999999e-06 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051242+0000", 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6988999999999999e-05 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051255+0000", 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3261999999999999e-05 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162787+0000", 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111532033 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.162923+0000", 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013530399999999999 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163156+0000", 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233095 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131616 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.647493506000004, 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112285617, 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131616 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.051273+0000", 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058214+0000", 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069405279999999996 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058226+0000", 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1732999999999999e-05 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058253+0000", 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7574999999999999e-05 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058276+0000", 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2334999999999999e-05 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163207+0000", 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104930906 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163336+0000", 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000129639 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.163559+0000", 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022290100000000001 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131631 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1220608~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.639864998, 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111259443, 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131631 2024-09-04T21:56:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058902+0000", 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058903+0000", 2024-09-04T21:56:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.133e-06 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058909+0000", 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6890000000000001e-06 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058926+0000", 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7166000000000001e-05 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058943+0000", 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7028999999999999e-05 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169875+0000", 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11093157400000001 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.169996+0000", 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012140499999999999 2024-09-04T21:56:13.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170161+0000", 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016544699999999999 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131632 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1613824~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.639802068999998, 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111614907, 2024-09-04T21:56:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131632 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058965+0000", 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058966+0000", 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1429999999999999e-06 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.058974+0000", 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9559999999999997e-06 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059013+0000", 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8688e-05 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059027+0000", 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4649e-05 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170250+0000", 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111222396 2024-09-04T21:56:13.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170381+0000", 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013136800000000001 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170580+0000", 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019870700000000001 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131633 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1703936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.639715055000003, 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11190109299999999, 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131633 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059052+0000", 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059053+0000", 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.353e-06 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059067+0000", 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3482999999999999e-05 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059104+0000", 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7657999999999998e-05 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059129+0000", 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4380999999999999e-05 2024-09-04T21:56:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170639+0000", 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111510427 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170761+0000", 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000122302 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.170953+0000", 2024-09-04T21:56:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019148900000000001 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131634 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1310720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.639615135, 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11221152299999999, 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131634 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059152+0000", 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059154+0000", 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059160+0000", 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4330000000000001e-06 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059199+0000", 2024-09-04T21:56:13.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9190999999999998e-05 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059220+0000", 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0499000000000001e-05 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171022+0000", 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11180261599999999 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171158+0000", 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135401 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171363+0000", 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020553999999999999 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131635 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 1794048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:13.782 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.639525673999998, 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.112472239, 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131635 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:13.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.059241+0000", 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066220+0000", 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0069786229999999998 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066229+0000", 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8060000000000001e-06 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066251+0000", 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1829000000000001e-05 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.066273+0000", 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2722e-05 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171426+0000", 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10515273 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171548+0000", 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012224 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.171714+0000", 2024-09-04T21:56:13.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000165289 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:131652 3.1a 3:5d903571:::rbd_data.1188363984ef.0000000000000000:1f [read 2785280~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.631635385999999, 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.111334216, 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:13.789 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 131652 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.067132+0000", 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:13.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074601+0000", 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0074699279999999998 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074617+0000", 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5882000000000001e-05 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074651+0000", 2024-09-04T21:56:13.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4100999999999998e-05 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.074674+0000", 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2858999999999999e-05 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178090+0000", 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10341562999999999 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:13.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178247+0000", 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156673 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:55:22.178466+0000", 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021914300000000001 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:13.793 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:13.795 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:56:13.795 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:13.790+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:13.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-04T21:56:13.896 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:13.890+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:13.946 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:13.946 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:13.946 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-04T21:56:13.946 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:13.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:56:13.997 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-09-04T21:56:13.995+0000 7f637eb2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35652) UID: 0 2024-09-04T21:56:14.098 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:14.094+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:14.144 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:14.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-04T21:56:14.144 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-04T21:56:14.144 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-04T21:56:14.144 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:14.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:56:14.198 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:14.195+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:14.300 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:14.299+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:14.316 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:14.316 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:32548 3.15 3:a86a8f56:::rbd_data.1188e6876e06.0000000000000003:f [read 7331840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.811088645, 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063243635000000006, 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.317 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 32548 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502842+0000", 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502843+0000", 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4499999999999996e-07 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502876+0000", 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2848000000000001e-05 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502888+0000", 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2554999999999999e-05 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.502906+0000", 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7510999999999999e-05 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565474+0000", 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062568718999999995 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.565788+0000", 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031323800000000003 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:30.566086+0000", 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029801900000000001 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49708 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 573440~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466312259000006, 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062916446000000001, 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.322 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49708 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847618+0000", 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847619+0000", 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5699999999999998e-07 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847628+0000", 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.312e-06 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847634+0000", 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3300000000000004e-06 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847639+0000", 2024-09-04T21:56:14.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7180000000000004e-06 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910239+0000", 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062599501000000002 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910293+0000", 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3976e-05 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910535+0000", 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024225200000000001 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.327 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49709 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 737280~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466283653999994, 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063352931000000001, 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49709 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.328 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847647+0000", 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847648+0000", 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4299999999999998e-07 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847650+0000", 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0470000000000001e-06 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847655+0000", 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9050000000000002e-06 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847660+0000", 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6020000000000002e-06 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910577+0000", 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.062917127000000003 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.910674+0000", 2024-09-04T21:56:14.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6351e-05 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911000+0000", 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032645599999999998 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49710 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 901120~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:56:14.332 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466263616999996, 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063653213, 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49710 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847667+0000", 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847668+0000", 2024-09-04T21:56:14.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9499999999999998e-07 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847669+0000", 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.905e-06 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847675+0000", 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7819999999999999e-06 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847680+0000", 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7430000000000002e-06 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911035+0000", 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063355380000000003 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911090+0000", 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4410000000000003e-05 2024-09-04T21:56:14.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911320+0000", 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023059799999999999 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49711 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1064960~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:56:14.337 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466244395000004, 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064044326999999998, 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49711 2024-09-04T21:56:14.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847686+0000", 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847687+0000", 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5100000000000001e-07 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847688+0000", 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.782e-06 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847694+0000", 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.925e-06 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847699+0000", 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.481e-06 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911362+0000", 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.063663202000000002 2024-09-04T21:56:14.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911461+0000", 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8474000000000006e-05 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911731+0000", 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027011199999999998 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49712 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1228800~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466225604000002, 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064364710000000006, 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.343 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49712 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847705+0000", 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847706+0000", 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4000000000000001e-07 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847708+0000", 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.843e-06 2024-09-04T21:56:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847714+0000", 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0639999999999999e-06 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847718+0000", 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911764+0000", 2024-09-04T21:56:14.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064045721999999999 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.911820+0000", 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5875999999999998e-05 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912070+0000", 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00025000800000000002 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49713 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1392640~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466193661999995, 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064791654000000004, 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.348 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49713 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847737+0000", 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847738+0000", 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6199999999999997e-07 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847744+0000", 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2530000000000001e-06 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847751+0000", 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2790000000000001e-06 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847757+0000", 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2230000000000004e-06 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912114+0000", 2024-09-04T21:56:14.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064356226000000002 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912208+0000", 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4269000000000002e-05 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912529+0000", 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000320742 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49714 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1556480~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466165133000004, 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065045087000000001, 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.353 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49714 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847766+0000", 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4900000000000001e-07 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847768+0000", 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.165e-06 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847775+0000", 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6059999999999999e-06 2024-09-04T21:56:14.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847779+0000", 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2189999999999998e-06 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912564+0000", 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.064784515000000001 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912617+0000", 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3847000000000001e-05 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912811+0000", 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019328600000000001 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.357 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49715 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1720320~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466145522000005, 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065489191000000002, 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49715 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847785+0000", 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847786+0000", 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9200000000000001e-07 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847788+0000", 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9290000000000001e-06 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847793+0000", 2024-09-04T21:56:14.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8730000000000004e-06 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847798+0000", 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.313e-06 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912856+0000", 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065058267000000003 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.912953+0000", 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7231999999999993e-05 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913274+0000", 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032108499999999998 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49716 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 638976~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.362 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466127228999994, 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065795943999999995, 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49716 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847803+0000", 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847804+0000", 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5199999999999997e-07 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847806+0000", 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.841e-06 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847811+0000", 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5090000000000001e-06 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847818+0000", 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.708e-06 2024-09-04T21:56:14.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913312+0000", 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065494259999999999 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913366+0000", 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3205999999999998e-05 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913599+0000", 2024-09-04T21:56:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000233868 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49717 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 802816~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466104415000004, 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183335999999995, 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49717 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847826+0000", 2024-09-04T21:56:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847827+0000", 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1500000000000004e-07 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847829+0000", 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.885e-06 2024-09-04T21:56:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847834+0000", 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5890000000000002e-06 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847840+0000", 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1070000000000004e-06 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913644+0000", 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.065804118999999994 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.913742+0000", 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7924999999999995e-05 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914010+0000", 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000268096 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49718 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 966656~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466085277999994, 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066502624999999996, 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49718 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847845+0000", 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847847+0000", 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2389999999999999e-06 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847849+0000", 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8649999999999999e-06 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847855+0000", 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0979999999999999e-06 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847860+0000", 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9019999999999998e-06 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914043+0000", 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066183097999999996 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914107+0000", 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4602000000000003e-05 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914348+0000", 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000240821 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49719 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1130496~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466064611999997, 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066949649999999999, 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49719 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847866+0000", 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847867+0000", 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2799999999999996e-07 2024-09-04T21:56:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847877+0000", 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8949999999999993e-06 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847885+0000", 2024-09-04T21:56:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7129999999999994e-06 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847890+0000", 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5569999999999997e-06 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914394+0000", 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066504092000000001 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914493+0000", 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8919999999999995e-05 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914816+0000", 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00032294499999999999 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49720 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1294336~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466033835000005, 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067244427999999995, 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49720 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847897+0000", 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2300000000000002e-07 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847899+0000", 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.995e-06 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847905+0000", 2024-09-04T21:56:14.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1840000000000004e-06 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847910+0000", 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4340000000000002e-06 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914850+0000", 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.066939632999999998 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.914905+0000", 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5430999999999998e-05 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915141+0000", 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000236328 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49721 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1458176~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:56:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.466014466999994, 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067632327000000006, 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49721 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:56:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847916+0000", 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847917+0000", 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6800000000000001e-07 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847920+0000", 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5470000000000001e-06 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847927+0000", 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3969999999999999e-06 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847932+0000", 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8710000000000002e-06 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915186+0000", 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067253755999999998 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915281+0000", 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4437000000000004e-05 2024-09-04T21:56:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915549+0000", 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000267951 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49722 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1622016~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.465990508000004, 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067890907, 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49722 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847940+0000", 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847941+0000", 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847943+0000", 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9570000000000001e-06 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847950+0000", 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6469999999999998e-06 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847955+0000", 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0529999999999997e-06 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915583+0000", 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067627408999999999 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915637+0000", 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4153000000000001e-05 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915831+0000", 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000194067 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49723 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1785856~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.465943258999999, 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068255171000000003, 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49723 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847987+0000", 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847988+0000", 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6400000000000002e-07 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847991+0000", 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5119999999999998e-06 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.847999+0000", 2024-09-04T21:56:14.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4060000000000005e-06 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848005+0000", 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1210000000000003e-06 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915875+0000", 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.067869437000000005 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.915972+0000", 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7542999999999998e-05 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916243+0000", 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027058799999999999 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49724 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1884160~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.400 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.465917580999999, 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068553664, 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.401 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49724 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848013+0000", 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848014+0000", 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7700000000000004e-07 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848016+0000", 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2180000000000002e-06 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848023+0000", 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.554e-06 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848029+0000", 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0220000000000004e-06 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916278+0000", 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068249803999999997 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916332+0000", 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3780999999999999e-05 2024-09-04T21:56:14.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916567+0000", 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000234708 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49725 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 2048000~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.465883555000005, 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068928329999999996, 2024-09-04T21:56:14.405 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49725 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848047+0000", 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848048+0000", 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848050+0000", 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5359999999999999e-06 2024-09-04T21:56:14.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848058+0000", 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6199999999999999e-06 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848063+0000", 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3120000000000001e-06 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916614+0000", 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068550875999999997 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916709+0000", 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5035000000000001e-05 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.916976+0000", 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026643800000000003 2024-09-04T21:56:14.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4488.0:49726 3.5 3:a352a6c5:::rbd_data.1188359b1b3c.0000000000000003:10 [read 1949696~8192] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e62)", 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.465859154, 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.069173545000000003, 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4488", 2024-09-04T21:56:14.410 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "172.21.15.112:0/903042047", 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49726 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-04T21:56:14.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848072+0000", 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5499999999999998e-07 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848074+0000", 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3659999999999999e-06 2024-09-04T21:56:14.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848082+0000", 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4780000000000002e-06 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.848087+0000", 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.468e-06 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-04T21:56:14.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917003+0000", 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.068915287000000006 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917058+0000", 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4982000000000003e-05 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-04T21:54:45.917245+0000", 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018740900000000001 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.414 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-04T21:56:14.415 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-04T21:56:14.415 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:14.415 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-04T21:56:14.417 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:56:14.417 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:14.410+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:14.517 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:14.510+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:14.561 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:56:14.561 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:56:14.561 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-04T21:56:14.561 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:56:14.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:56:14.619 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:14.619+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:14.622 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0: {} 2024-09-04T21:56:14.622 INFO:teuthology.orchestra.run.smithi112.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-04T21:56:14.639 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1: {} 2024-09-04T21:56:14.639 INFO:teuthology.orchestra.run.smithi112.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-04T21:56:14.656 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2: {} 2024-09-04T21:56:14.656 INFO:teuthology.orchestra.run.smithi112.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-04T21:56:14.672 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3: {} 2024-09-04T21:56:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-04T21:56:14.691 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4: {} 2024-09-04T21:56:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-04T21:56:14.702 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-04T21:56:14.702 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-04T21:56:14.703 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-04T21:56:14.703 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-04T21:56:14.703 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-04T21:56:14.712 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5: {} 2024-09-04T21:56:14.712 INFO:teuthology.orchestra.run.smithi112.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-04T21:56:14.718 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:56:14.733 INFO:teuthology.orchestra.run.smithi112.stdout:osd.6: {} 2024-09-04T21:56:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-04T21:56:14.752 INFO:teuthology.orchestra.run.smithi112.stdout:osd.7: {} 2024-09-04T21:56:14.752 INFO:teuthology.orchestra.run.smithi112.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-04T21:56:14.760 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:14.759+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:14.820 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:14.814+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:14.878 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:14.878 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:14.892 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:56:14.935 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:14.935+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:15.015 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:15.015 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:15.021 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:15.019+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:15.030 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:56:15.122 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-04T21:56:15.118+0000 7fbec4de2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26418) UID: 0 2024-09-04T21:56:15.195 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:15.195 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:15.209 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:56:15.224 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:15.222+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:15.324 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-04T21:56:15.323+0000 7fb397376640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32195) UID: 0 2024-09-04T21:56:15.333 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:15.333 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:15.347 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:56:15.426 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:15.422+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:15.483 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:56:15.526 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:15.523+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:15.532 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:15.532 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:15.546 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:56:15.631 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:15.631+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:15.714 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:15.714 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:15.728 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:15.727+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:15.729 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:56:15.829 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:15.827+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:15.889 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:15.890 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:15.904 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:56:15.929 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:15.926+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:16.034 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:16.031+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:16.064 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.064 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:16.078 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:56:16.134 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:16.130+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:16.235 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:16.234+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:16.236 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.236 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:16.251 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:56:16.337 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-09-04T21:56:16.335+0000 7f9f5c8c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26495) UID: 0 2024-09-04T21:56:16.375 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.375 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:16.389 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-04T21:56:16.438 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-09-04T21:56:16.434+0000 7f6290f5a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26421) UID: 0 2024-09-04T21:56:16.487 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:56:16.534 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.534 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:16.539 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-04T21:56:16.534+0000 7f97d2609640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26417) UID: 0 2024-09-04T21:56:16.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-04T21:56:16.621 INFO:teuthology.orchestra.run.smithi112.stdout:All operations completed A-OK! 2024-09-04T21:56:16.640 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-09-04T21:56:16.639+0000 7fd058662640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26496) UID: 0 2024-09-04T21:56:16.664 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:16.664 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-09-04T21:56:16.676 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-04T21:56:16.685 INFO:tasks.thrashosds:joining thrashosds 2024-09-04T21:56:16.715 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.715 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:16.730 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-04T21:56:16.852 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.852 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:16.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T21:56:16.986 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:16.987 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.002 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-04T21:56:17.122 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.123 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-04T21:56:17.253 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.253 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-04T21:56:17.385 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.385 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.399 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-04T21:56:17.518 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.518 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.532 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-04T21:56:17.649 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.650 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-04T21:56:17.788 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.788 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.802 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-04T21:56:17.924 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:17.925 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:17.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-04T21:56:18.059 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.059 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.073 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-04T21:56:18.201 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.202 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.216 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-04T21:56:18.334 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.335 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.349 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-04T21:56:18.474 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.474 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.489 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-04T21:56:18.516 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-04T21:56:18.616 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.616 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.630 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-04T21:56:18.755 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.756 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.770 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-04T21:56:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:18.890 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:18.904 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-04T21:56:18.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-04T21:56:19.030 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:19.030 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:19.044 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-04T21:56:19.140 INFO:teuthology.orchestra.run.smithi112.stdout:mon.a: {} 2024-09-04T21:56:19.140 INFO:teuthology.orchestra.run.smithi112.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-04T21:56:19.152 INFO:teuthology.orchestra.run.smithi112.stdout:mon.b: {} 2024-09-04T21:56:19.152 INFO:teuthology.orchestra.run.smithi112.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-04T21:56:19.161 INFO:teuthology.orchestra.run.smithi112.stdout:mon.c: {} 2024-09-04T21:56:19.161 INFO:teuthology.orchestra.run.smithi112.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-04T21:56:19.162 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:19.162 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:19.176 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-04T21:56:19.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-04T21:56:19.296 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:19.297 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:19.311 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-04T21:56:19.429 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:19.429 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:19.443 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-04T21:56:19.500 INFO:teuthology.orchestra.run.smithi112.stderr:noscrub is unset 2024-09-04T21:56:19.500 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.0 to 1 (10000) 2024-09-04T21:56:19.525 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-04T21:56:19.526 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-04T21:56:19.567 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:19.567 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:19.582 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-04T21:56:19.705 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-04T21:56:19.705 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-04T21:56:20.371 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-04T21:56:20.371 INFO:teuthology.orchestra.run.smithi112.stderr:nodeep-scrub is unset 2024-09-04T21:56:20.395 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-04T21:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.1 to 1 (10000) 2024-09-04T21:56:21.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-04T21:56:22.536 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-04T21:56:22.564 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-04T21:56:23.538 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.2 to 1 (10000) 2024-09-04T21:56:23.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-04T21:56:24.550 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-04T21:56:24.578 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-04T21:56:25.367 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.3 to 1 (10000) 2024-09-04T21:56:25.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-04T21:56:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-04T21:56:26.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-04T21:56:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.5 to 1 (10000) 2024-09-04T21:56:27.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-04T21:56:28.582 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-04T21:56:28.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-04T21:56:29.606 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.6 to 1 (10000) 2024-09-04T21:56:29.630 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-04T21:56:30.602 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-04T21:56:30.630 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-04T21:56:31.619 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.7 to 1 (10000) 2024-09-04T21:56:31.644 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-04T21:56:32.635 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-04T21:56:32.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-04T21:56:33.639 INFO:teuthology.orchestra.run.smithi112.stderr:reweighted osd.4 to 1 (10000) 2024-09-04T21:56:33.666 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-04T21:56:34.649 INFO:teuthology.orchestra.run.smithi112.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-04T21:56:34.674 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-04T21:56:34.674 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-04T21:56:34.674 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-04T21:56:34.674 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-04T21:56:34.675 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-04T21:56:34.675 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-04T21:56:34.970 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:34.970 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":160,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:56:34.646237+0000","last_up_change":"2024-09-04T21:55:37.706985+0000","last_in_change":"2024-09-04T21:56:00.354804+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-04T21:53:36.459454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-04T21:53:43.849962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-04T21:53:54.481634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":36,"snap_epoch":144,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6817","nonce":3808224845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6819","nonce":3808224845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6823","nonce":3808224845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6821","nonce":3808224845}]},"public_addr":"172.21.15.112:6817/3808224845","cluster_addr":"172.21.15.112:6819/3808224845","heartbeat_back_addr":"172.21.15.112:6823/3808224845","heartbeat_front_addr":"172.21.15.112:6821/3808224845","state":["exists","up"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6825","nonce":245081828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6827","nonce":245081828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6831","nonce":245081828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6829","nonce":245081828}]},"public_addr":"172.21.15.112:6825/245081828","cluster_addr":"172.21.15.112:6827/245081828","heartbeat_back_addr":"172.21.15.112:6831/245081828","heartbeat_front_addr":"172.21.15.112:6829/245081828","state":["exists","up"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6801","nonce":1590148387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6803","nonce":1590148387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6807","nonce":1590148387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6805","nonce":1590148387}]},"public_addr":"172.21.15.112:6801/1590148387","cluster_addr":"172.21.15.112:6803/1590148387","heartbeat_back_addr":"172.21.15.112:6807/1590148387","heartbeat_front_addr":"172.21.15.112:6805/1590148387","state":["exists","up"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6809","nonce":2461796488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6811","nonce":2461796488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6815","nonce":2461796488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6813","nonce":2461796488}]},"public_addr":"172.21.15.112:6809/2461796488","cluster_addr":"172.21.15.112:6811/2461796488","heartbeat_back_addr":"172.21.15.112:6815/2461796488","heartbeat_front_addr":"172.21.15.112:6813/2461796488","state":["exists","up"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":84,"up_from":105,"up_thru":137,"down_at":85,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6825","nonce":3575407730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6827","nonce":3575407730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6831","nonce":3575407730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6829","nonce":3575407730}]},"public_addr":"172.21.15.157:6825/3575407730","cluster_addr":"172.21.15.157:6827/3575407730","heartbeat_back_addr":"172.21.15.157:6831/3575407730","heartbeat_front_addr":"172.21.15.157:6829/3575407730","state":["exists","up"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6801","nonce":3465983375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6803","nonce":3465983375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6808","nonce":3465983375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6805","nonce":3465983375}]},"public_addr":"172.21.15.157:6801/3465983375","cluster_addr":"172.21.15.157:6803/3465983375","heartbeat_back_addr":"172.21.15.157:6808/3465983375","heartbeat_front_addr":"172.21.15.157:6805/3465983375","state":["exists","up"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":21,"up_from":48,"up_thru":128,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6817","nonce":471611328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6819","nonce":471611328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6823","nonce":471611328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6821","nonce":471611328}]},"public_addr":"172.21.15.157:6817/471611328","cluster_addr":"172.21.15.157:6819/471611328","heartbeat_back_addr":"172.21.15.157:6823/471611328","heartbeat_front_addr":"172.21.15.157:6821/471611328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.725388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:35.420675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.389632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.611983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.796105+0000","dead_epoch":85},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.383432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.731214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.469040+0000","dead_epoch":22}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:56:34.988 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-04T21:56:34.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-04T21:56:35.276 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:35.276 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":160,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:56:34.646237+0000","last_up_change":"2024-09-04T21:55:37.706985+0000","last_in_change":"2024-09-04T21:56:00.354804+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-04T21:53:36.459454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-04T21:53:43.849962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-04T21:53:54.481634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":36,"snap_epoch":144,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6817","nonce":3808224845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6819","nonce":3808224845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6823","nonce":3808224845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6821","nonce":3808224845}]},"public_addr":"172.21.15.112:6817/3808224845","cluster_addr":"172.21.15.112:6819/3808224845","heartbeat_back_addr":"172.21.15.112:6823/3808224845","heartbeat_front_addr":"172.21.15.112:6821/3808224845","state":["exists","up"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6825","nonce":245081828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6827","nonce":245081828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6831","nonce":245081828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6829","nonce":245081828}]},"public_addr":"172.21.15.112:6825/245081828","cluster_addr":"172.21.15.112:6827/245081828","heartbeat_back_addr":"172.21.15.112:6831/245081828","heartbeat_front_addr":"172.21.15.112:6829/245081828","state":["exists","up"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6801","nonce":1590148387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6803","nonce":1590148387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6807","nonce":1590148387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6805","nonce":1590148387}]},"public_addr":"172.21.15.112:6801/1590148387","cluster_addr":"172.21.15.112:6803/1590148387","heartbeat_back_addr":"172.21.15.112:6807/1590148387","heartbeat_front_addr":"172.21.15.112:6805/1590148387","state":["exists","up"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6809","nonce":2461796488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6811","nonce":2461796488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6815","nonce":2461796488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6813","nonce":2461796488}]},"public_addr":"172.21.15.112:6809/2461796488","cluster_addr":"172.21.15.112:6811/2461796488","heartbeat_back_addr":"172.21.15.112:6815/2461796488","heartbeat_front_addr":"172.21.15.112:6813/2461796488","state":["exists","up"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":84,"up_from":105,"up_thru":137,"down_at":85,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6825","nonce":3575407730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6827","nonce":3575407730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6831","nonce":3575407730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6829","nonce":3575407730}]},"public_addr":"172.21.15.157:6825/3575407730","cluster_addr":"172.21.15.157:6827/3575407730","heartbeat_back_addr":"172.21.15.157:6831/3575407730","heartbeat_front_addr":"172.21.15.157:6829/3575407730","state":["exists","up"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6801","nonce":3465983375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6803","nonce":3465983375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6808","nonce":3465983375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6805","nonce":3465983375}]},"public_addr":"172.21.15.157:6801/3465983375","cluster_addr":"172.21.15.157:6803/3465983375","heartbeat_back_addr":"172.21.15.157:6808/3465983375","heartbeat_front_addr":"172.21.15.157:6805/3465983375","state":["exists","up"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":21,"up_from":48,"up_thru":128,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6817","nonce":471611328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6819","nonce":471611328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6823","nonce":471611328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6821","nonce":471611328}]},"public_addr":"172.21.15.157:6817/471611328","cluster_addr":"172.21.15.157:6819/471611328","heartbeat_back_addr":"172.21.15.157:6823/471611328","heartbeat_front_addr":"172.21.15.157:6821/471611328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.725388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:35.420675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.389632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.611983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.796105+0000","dead_epoch":85},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.383432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.731214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.469040+0000","dead_epoch":22}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:56:35.295 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-04T21:56:35.295 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-04T21:56:35.296 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-04T21:56:35.296 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-04T21:56:35.296 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-04T21:56:35.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-04T21:56:35.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-04T21:56:35.298 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-04T21:56:35.542 INFO:teuthology.orchestra.run.smithi112.stdout:64424509481 2024-09-04T21:56:35.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-04T21:56:35.560 INFO:teuthology.orchestra.run.smithi112.stdout:60129542184 2024-09-04T21:56:35.560 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-04T21:56:35.562 INFO:teuthology.orchestra.run.smithi112.stdout:206158430235 2024-09-04T21:56:35.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-04T21:56:35.566 INFO:teuthology.orchestra.run.smithi112.stdout:55834574887 2024-09-04T21:56:35.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-04T21:56:35.569 INFO:teuthology.orchestra.run.smithi112.stdout:450971566093 2024-09-04T21:56:35.569 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-04T21:56:35.570 INFO:teuthology.orchestra.run.smithi112.stdout:60129542183 2024-09-04T21:56:35.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-04T21:56:35.572 INFO:teuthology.orchestra.run.smithi112.stdout:64424509480 2024-09-04T21:56:35.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-04T21:56:35.580 INFO:teuthology.orchestra.run.smithi112.stdout:64424509480 2024-09-04T21:56:35.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-04T21:56:36.064 INFO:teuthology.orchestra.run.smithi112.stdout:206158430233 2024-09-04T21:56:36.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 206158430235 got 206158430233 for osd.7 2024-09-04T21:56:36.097 INFO:teuthology.orchestra.run.smithi112.stdout:64424509480 2024-09-04T21:56:36.097 INFO:teuthology.orchestra.run.smithi112.stdout:55834574886 2024-09-04T21:56:36.100 INFO:teuthology.orchestra.run.smithi112.stdout:60129542182 2024-09-04T21:56:36.102 INFO:teuthology.orchestra.run.smithi112.stdout:60129542183 2024-09-04T21:56:36.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509480 for osd.0 2024-09-04T21:56:36.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.5 2024-09-04T21:56:36.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542183 got 60129542182 for osd.3 2024-09-04T21:56:36.136 INFO:teuthology.orchestra.run.smithi112.stdout:450971566092 2024-09-04T21:56:36.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542183 for osd.2 2024-09-04T21:56:36.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 450971566093 got 450971566092 for osd.4 2024-09-04T21:56:36.165 INFO:teuthology.orchestra.run.smithi112.stdout:64424509479 2024-09-04T21:56:36.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509479 for osd.6 2024-09-04T21:56:36.189 INFO:teuthology.orchestra.run.smithi112.stdout:64424509479 2024-09-04T21:56:36.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509479 for osd.1 2024-09-04T21:56:37.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-04T21:56:37.123 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-04T21:56:37.131 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-04T21:56:37.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-04T21:56:37.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-04T21:56:37.154 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-04T21:56:37.189 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-04T21:56:37.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-04T21:56:37.608 INFO:teuthology.orchestra.run.smithi112.stdout:206158430235 2024-09-04T21:56:37.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 206158430235 got 206158430235 for osd.7 2024-09-04T21:56:37.634 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.642 INFO:teuthology.orchestra.run.smithi112.stdout:55834574887 2024-09-04T21:56:37.644 INFO:teuthology.orchestra.run.smithi112.stdout:60129542183 2024-09-04T21:56:37.664 INFO:teuthology.orchestra.run.smithi112.stdout:64424509481 2024-09-04T21:56:37.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542183 got 60129542183 for osd.3 2024-09-04T21:56:37.666 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.674 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.5 2024-09-04T21:56:37.674 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509481 for osd.0 2024-09-04T21:56:37.691 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.691 INFO:teuthology.orchestra.run.smithi112.stdout:60129542184 2024-09-04T21:56:37.716 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542184 for osd.2 2024-09-04T21:56:37.716 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.724 INFO:teuthology.orchestra.run.smithi112.stdout:64424509480 2024-09-04T21:56:37.729 INFO:teuthology.orchestra.run.smithi112.stdout:450971566093 2024-09-04T21:56:37.734 INFO:teuthology.orchestra.run.smithi112.stdout:64424509480 2024-09-04T21:56:37.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509480 for osd.6 2024-09-04T21:56:37.746 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 450971566093 got 450971566093 for osd.4 2024-09-04T21:56:37.756 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509480 for osd.1 2024-09-04T21:56:37.760 DEBUG:teuthology.parallel:result is None 2024-09-04T21:56:37.760 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-04T21:56:37.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:38.080 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:38.096 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-04T21:56:36.472190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355020,"kb_used_data":15040,"kb_used_omap":46,"kb_used_meta":339729,"kb_avail":754619700,"statfs":{"total":773094113280,"available":772730572800,"internally_reserved":0,"allocated":15400960,"data_stored":8513764,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":347882642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.851584"},"pg_stats":[{"pgid":"3.1f","version":"144'764","reported_seq":2993,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352518+0000","last_change":"2024-09-04T21:56:16.529040+0000","last_active":"2024-09-04T21:56:35.352518+0000","last_peered":"2024-09-04T21:56:35.352518+0000","last_clean":"2024-09-04T21:56:35.352518+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:35.352518+0000","last_undegraded":"2024-09-04T21:56:35.352518+0000","last_fullsized":"2024-09-04T21:56:35.352518+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:39:44.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"143'382","reported_seq":3537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653287+0000","last_change":"2024-09-04T21:56:16.490790+0000","last_active":"2024-09-04T21:56:34.653287+0000","last_peered":"2024-09-04T21:56:34.653287+0000","last_clean":"2024-09-04T21:56:34.653287+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:34.653287+0000","last_undegraded":"2024-09-04T21:56:34.653287+0000","last_fullsized":"2024-09-04T21:56:34.653287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:56:20.741571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"143'566","reported_seq":19161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861349+0000","last_change":"2024-09-04T21:56:16.556682+0000","last_active":"2024-09-04T21:56:34.861349+0000","last_peered":"2024-09-04T21:56:34.861349+0000","last_clean":"2024-09-04T21:56:34.861349+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:34.861349+0000","last_undegraded":"2024-09-04T21:56:34.861349+0000","last_fullsized":"2024-09-04T21:56:34.861349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:18:12.735862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"144'394","reported_seq":2484,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182638+0000","last_change":"2024-09-04T21:56:29.616344+0000","last_active":"2024-09-04T21:56:35.182638+0000","last_peered":"2024-09-04T21:56:35.182638+0000","last_clean":"2024-09-04T21:56:35.182638+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:35.182638+0000","last_undegraded":"2024-09-04T21:56:35.182638+0000","last_fullsized":"2024-09-04T21:56:35.182638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:43:32.577364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"143'290","reported_seq":1711,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352394+0000","last_change":"2024-09-04T21:56:16.527857+0000","last_active":"2024-09-04T21:56:35.352394+0000","last_peered":"2024-09-04T21:56:35.352394+0000","last_clean":"2024-09-04T21:56:35.352394+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:35.352394+0000","last_undegraded":"2024-09-04T21:56:35.352394+0000","last_fullsized":"2024-09-04T21:56:35.352394+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:15:56.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"143'957","reported_seq":17593,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652386+0000","last_change":"2024-09-04T21:56:16.488214+0000","last_active":"2024-09-04T21:56:34.652386+0000","last_peered":"2024-09-04T21:56:34.652386+0000","last_clean":"2024-09-04T21:56:34.652386+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:34.652386+0000","last_undegraded":"2024-09-04T21:56:34.652386+0000","last_fullsized":"2024-09-04T21:56:34.652386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:19.822098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"143'1213","reported_seq":3290,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654113+0000","last_change":"2024-09-04T21:56:16.491514+0000","last_active":"2024-09-04T21:56:34.654113+0000","last_peered":"2024-09-04T21:56:34.654113+0000","last_clean":"2024-09-04T21:56:34.654113+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:34.654113+0000","last_undegraded":"2024-09-04T21:56:34.654113+0000","last_fullsized":"2024-09-04T21:56:34.654113+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:21.020845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'402","reported_seq":4769,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653256+0000","last_change":"2024-09-04T21:56:16.488004+0000","last_active":"2024-09-04T21:56:34.653256+0000","last_peered":"2024-09-04T21:56:34.653256+0000","last_clean":"2024-09-04T21:56:34.653256+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:34.653256+0000","last_undegraded":"2024-09-04T21:56:34.653256+0000","last_fullsized":"2024-09-04T21:56:34.653256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:00.780840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"144'956","reported_seq":3245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352481+0000","last_change":"2024-09-04T21:56:16.528237+0000","last_active":"2024-09-04T21:56:35.352481+0000","last_peered":"2024-09-04T21:56:35.352481+0000","last_clean":"2024-09-04T21:56:35.352481+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:35.352481+0000","last_undegraded":"2024-09-04T21:56:35.352481+0000","last_fullsized":"2024-09-04T21:56:35.352481+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:49:16.916644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'357","reported_seq":2708,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.183014+0000","last_change":"2024-09-04T21:56:29.617384+0000","last_active":"2024-09-04T21:56:35.183014+0000","last_peered":"2024-09-04T21:56:35.183014+0000","last_clean":"2024-09-04T21:56:35.183014+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:35.183014+0000","last_undegraded":"2024-09-04T21:56:35.183014+0000","last_fullsized":"2024-09-04T21:56:35.183014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:30:08.502145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"144'393","reported_seq":3169,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227566+0000","last_change":"2024-09-04T21:56:16.600842+0000","last_active":"2024-09-04T21:56:35.227566+0000","last_peered":"2024-09-04T21:56:35.227566+0000","last_clean":"2024-09-04T21:56:35.227566+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:35.227566+0000","last_undegraded":"2024-09-04T21:56:35.227566+0000","last_fullsized":"2024-09-04T21:56:35.227566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:00.160039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"143'345","reported_seq":2075,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652357+0000","last_change":"2024-09-04T21:56:16.488308+0000","last_active":"2024-09-04T21:56:34.652357+0000","last_peered":"2024-09-04T21:56:34.652357+0000","last_clean":"2024-09-04T21:56:34.652357+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:34.652357+0000","last_undegraded":"2024-09-04T21:56:34.652357+0000","last_fullsized":"2024-09-04T21:56:34.652357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:24:16.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353101+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:35.353101+0000","last_peered":"2024-09-04T21:56:35.353101+0000","last_clean":"2024-09-04T21:56:35.353101+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:35.353101+0000","last_undegraded":"2024-09-04T21:56:35.353101+0000","last_fullsized":"2024-09-04T21:56:35.353101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"144'465","reported_seq":2549,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352500+0000","last_change":"2024-09-04T21:56:16.527617+0000","last_active":"2024-09-04T21:56:35.352500+0000","last_peered":"2024-09-04T21:56:35.352500+0000","last_clean":"2024-09-04T21:56:35.352500+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:35.352500+0000","last_undegraded":"2024-09-04T21:56:35.352500+0000","last_fullsized":"2024-09-04T21:56:35.352500+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:34:20.478729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654438+0000","last_change":"2024-09-04T21:55:50.373208+0000","last_active":"2024-09-04T21:56:34.654438+0000","last_peered":"2024-09-04T21:56:34.654438+0000","last_clean":"2024-09-04T21:56:34.654438+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:34.654438+0000","last_undegraded":"2024-09-04T21:56:34.654438+0000","last_fullsized":"2024-09-04T21:56:34.654438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:34.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"143'353","reported_seq":3575,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653090+0000","last_change":"2024-09-04T21:56:16.491647+0000","last_active":"2024-09-04T21:56:34.653090+0000","last_peered":"2024-09-04T21:56:34.653090+0000","last_clean":"2024-09-04T21:56:34.653090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:34.653090+0000","last_undegraded":"2024-09-04T21:56:34.653090+0000","last_fullsized":"2024-09-04T21:56:34.653090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:45:54.303973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227837+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:35.227837+0000","last_peered":"2024-09-04T21:56:35.227837+0000","last_clean":"2024-09-04T21:56:35.227837+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:35.227837+0000","last_undegraded":"2024-09-04T21:56:35.227837+0000","last_fullsized":"2024-09-04T21:56:35.227837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"143'330","reported_seq":1971,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654382+0000","last_change":"2024-09-04T21:56:16.492759+0000","last_active":"2024-09-04T21:56:34.654382+0000","last_peered":"2024-09-04T21:56:34.654382+0000","last_clean":"2024-09-04T21:56:34.654382+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:34.654382+0000","last_undegraded":"2024-09-04T21:56:34.654382+0000","last_fullsized":"2024-09-04T21:56:34.654382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:13:53.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654037+0000","last_change":"2024-09-04T21:56:01.380611+0000","last_active":"2024-09-04T21:56:34.654037+0000","last_peered":"2024-09-04T21:56:34.654037+0000","last_clean":"2024-09-04T21:56:34.654037+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:34.654037+0000","last_undegraded":"2024-09-04T21:56:34.654037+0000","last_fullsized":"2024-09-04T21:56:34.654037+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:44.108771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'560","reported_seq":11918,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182904+0000","last_change":"2024-09-04T21:56:29.614226+0000","last_active":"2024-09-04T21:56:35.182904+0000","last_peered":"2024-09-04T21:56:35.182904+0000","last_clean":"2024-09-04T21:56:35.182904+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:35.182904+0000","last_undegraded":"2024-09-04T21:56:35.182904+0000","last_fullsized":"2024-09-04T21:56:35.182904+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:09:39.718023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"144'509","reported_seq":4121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652418+0000","last_change":"2024-09-04T21:56:16.490969+0000","last_active":"2024-09-04T21:56:34.652418+0000","last_peered":"2024-09-04T21:56:34.652418+0000","last_clean":"2024-09-04T21:56:34.652418+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:34.652418+0000","last_undegraded":"2024-09-04T21:56:34.652418+0000","last_fullsized":"2024-09-04T21:56:34.652418+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:46:12.808214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182664+0000","last_change":"2024-09-04T21:56:29.610468+0000","last_active":"2024-09-04T21:56:35.182664+0000","last_peered":"2024-09-04T21:56:35.182664+0000","last_clean":"2024-09-04T21:56:35.182664+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:35.182664+0000","last_undegraded":"2024-09-04T21:56:35.182664+0000","last_fullsized":"2024-09-04T21:56:35.182664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:27:48.891402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"143'376","reported_seq":3305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653993+0000","last_change":"2024-09-04T21:56:16.493672+0000","last_active":"2024-09-04T21:56:34.653993+0000","last_peered":"2024-09-04T21:56:34.653993+0000","last_clean":"2024-09-04T21:56:34.653993+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:34.653993+0000","last_undegraded":"2024-09-04T21:56:34.653993+0000","last_fullsized":"2024-09-04T21:56:34.653993+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:44:32.875548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861461+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:56:34.861461+0000","last_peered":"2024-09-04T21:56:34.861461+0000","last_clean":"2024-09-04T21:56:34.861461+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:34.861461+0000","last_undegraded":"2024-09-04T21:56:34.861461+0000","last_fullsized":"2024-09-04T21:56:34.861461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'618","reported_seq":4691,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352208+0000","last_change":"2024-09-04T21:56:16.527386+0000","last_active":"2024-09-04T21:56:35.352208+0000","last_peered":"2024-09-04T21:56:35.352208+0000","last_clean":"2024-09-04T21:56:35.352208+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:35.352208+0000","last_undegraded":"2024-09-04T21:56:35.352208+0000","last_fullsized":"2024-09-04T21:56:35.352208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:25:55.616051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227578+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:35.227578+0000","last_peered":"2024-09-04T21:56:35.227578+0000","last_clean":"2024-09-04T21:56:35.227578+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:35.227578+0000","last_undegraded":"2024-09-04T21:56:35.227578+0000","last_fullsized":"2024-09-04T21:56:35.227578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"144'426","reported_seq":3648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652929+0000","last_change":"2024-09-04T21:56:16.487890+0000","last_active":"2024-09-04T21:56:34.652929+0000","last_peered":"2024-09-04T21:56:34.652929+0000","last_clean":"2024-09-04T21:56:34.652929+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:34.652929+0000","last_undegraded":"2024-09-04T21:56:34.652929+0000","last_fullsized":"2024-09-04T21:56:34.652929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:00:48.401698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":115,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182685+0000","last_change":"2024-09-04T21:56:29.613744+0000","last_active":"2024-09-04T21:56:35.182685+0000","last_peered":"2024-09-04T21:56:35.182685+0000","last_clean":"2024-09-04T21:56:35.182685+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:35.182685+0000","last_undegraded":"2024-09-04T21:56:35.182685+0000","last_fullsized":"2024-09-04T21:56:35.182685+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:26.686223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227261+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:35.227261+0000","last_peered":"2024-09-04T21:56:35.227261+0000","last_clean":"2024-09-04T21:56:35.227261+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:35.227261+0000","last_undegraded":"2024-09-04T21:56:35.227261+0000","last_fullsized":"2024-09-04T21:56:35.227261+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"144'590","reported_seq":3820,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652932+0000","last_change":"2024-09-04T21:56:16.489907+0000","last_active":"2024-09-04T21:56:34.652932+0000","last_peered":"2024-09-04T21:56:34.652932+0000","last_clean":"2024-09-04T21:56:34.652932+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:34.652932+0000","last_undegraded":"2024-09-04T21:56:34.652932+0000","last_fullsized":"2024-09-04T21:56:34.652932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:10:28.643680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"143'345","reported_seq":2203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652276+0000","last_change":"2024-09-04T21:56:16.491283+0000","last_active":"2024-09-04T21:56:34.652276+0000","last_peered":"2024-09-04T21:56:34.652276+0000","last_clean":"2024-09-04T21:56:34.652276+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:34.652276+0000","last_undegraded":"2024-09-04T21:56:34.652276+0000","last_fullsized":"2024-09-04T21:56:34.652276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:43:39.964868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"143'706","reported_seq":2977,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353002+0000","last_change":"2024-09-04T21:56:16.584413+0000","last_active":"2024-09-04T21:56:35.353002+0000","last_peered":"2024-09-04T21:56:35.353002+0000","last_clean":"2024-09-04T21:56:35.353002+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:35.353002+0000","last_undegraded":"2024-09-04T21:56:35.353002+0000","last_fullsized":"2024-09-04T21:56:35.353002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:01:21.655483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"143'383","reported_seq":2457,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653129+0000","last_change":"2024-09-04T21:56:16.488947+0000","last_active":"2024-09-04T21:56:34.653129+0000","last_peered":"2024-09-04T21:56:34.653129+0000","last_clean":"2024-09-04T21:56:34.653129+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:34.653129+0000","last_undegraded":"2024-09-04T21:56:34.653129+0000","last_fullsized":"2024-09-04T21:56:34.653129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:49.649633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"144'357","reported_seq":3597,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182710+0000","last_change":"2024-09-04T21:56:29.613277+0000","last_active":"2024-09-04T21:56:35.182710+0000","last_peered":"2024-09-04T21:56:35.182710+0000","last_clean":"2024-09-04T21:56:35.182710+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:35.182710+0000","last_undegraded":"2024-09-04T21:56:35.182710+0000","last_fullsized":"2024-09-04T21:56:35.182710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:00.762632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'419","reported_seq":11217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227512+0000","last_change":"2024-09-04T21:56:16.600636+0000","last_active":"2024-09-04T21:56:35.227512+0000","last_peered":"2024-09-04T21:56:35.227512+0000","last_clean":"2024-09-04T21:56:35.227512+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:35.227512+0000","last_undegraded":"2024-09-04T21:56:35.227512+0000","last_fullsized":"2024-09-04T21:56:35.227512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:53:20.412017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"143'394","reported_seq":1388,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227770+0000","last_change":"2024-09-04T21:56:16.601950+0000","last_active":"2024-09-04T21:56:35.227770+0000","last_peered":"2024-09-04T21:56:35.227770+0000","last_clean":"2024-09-04T21:56:35.227770+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:35.227770+0000","last_undegraded":"2024-09-04T21:56:35.227770+0000","last_fullsized":"2024-09-04T21:56:35.227770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:42.603261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"143'488","reported_seq":11950,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227095+0000","last_change":"2024-09-04T21:56:16.602325+0000","last_active":"2024-09-04T21:56:35.227095+0000","last_peered":"2024-09-04T21:56:35.227095+0000","last_clean":"2024-09-04T21:56:35.227095+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:35.227095+0000","last_undegraded":"2024-09-04T21:56:35.227095+0000","last_fullsized":"2024-09-04T21:56:35.227095+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:41:05.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"143'830","reported_seq":3709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352971+0000","last_change":"2024-09-04T21:56:16.584288+0000","last_active":"2024-09-04T21:56:35.352971+0000","last_peered":"2024-09-04T21:56:35.352971+0000","last_clean":"2024-09-04T21:56:35.352971+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:35.352971+0000","last_undegraded":"2024-09-04T21:56:35.352971+0000","last_fullsized":"2024-09-04T21:56:35.352971+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:03:59.014910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"144'579","reported_seq":6064,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227102+0000","last_change":"2024-09-04T21:56:16.598876+0000","last_active":"2024-09-04T21:56:35.227102+0000","last_peered":"2024-09-04T21:56:35.227102+0000","last_clean":"2024-09-04T21:56:35.227102+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:35.227102+0000","last_undegraded":"2024-09-04T21:56:35.227102+0000","last_fullsized":"2024-09-04T21:56:35.227102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:34:16.388010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"143'763","reported_seq":13635,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352207+0000","last_change":"2024-09-04T21:56:16.528688+0000","last_active":"2024-09-04T21:56:35.352207+0000","last_peered":"2024-09-04T21:56:35.352207+0000","last_clean":"2024-09-04T21:56:35.352207+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:35.352207+0000","last_undegraded":"2024-09-04T21:56:35.352207+0000","last_fullsized":"2024-09-04T21:56:35.352207+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:08.477339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"142'973","reported_seq":2644,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227285+0000","last_change":"2024-09-04T21:56:16.596597+0000","last_active":"2024-09-04T21:56:35.227285+0000","last_peered":"2024-09-04T21:56:35.227285+0000","last_clean":"2024-09-04T21:56:35.227285+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:35.227285+0000","last_undegraded":"2024-09-04T21:56:35.227285+0000","last_fullsized":"2024-09-04T21:56:35.227285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:19:55.348359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17483,"ondisk_log_size":17483,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566093,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39276,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332564,"statfs":{"total":96636764160,"available":96596545536,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1945600,"data_stored":1504066,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542183,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45532,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326308,"statfs":{"total":96636764160,"available":96590139392,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34072,"kb_used_data":1916,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337768,"statfs":{"total":96636764160,"available":96601874432,"internally_reserved":0,"allocated":1961984,"data_stored":1504066,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:38.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:38.390 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:38.406 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-04T21:56:36.472190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355020,"kb_used_data":15040,"kb_used_omap":46,"kb_used_meta":339729,"kb_avail":754619700,"statfs":{"total":773094113280,"available":772730572800,"internally_reserved":0,"allocated":15400960,"data_stored":8513764,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":347882642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.851584"},"pg_stats":[{"pgid":"3.1f","version":"144'764","reported_seq":2993,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352518+0000","last_change":"2024-09-04T21:56:16.529040+0000","last_active":"2024-09-04T21:56:35.352518+0000","last_peered":"2024-09-04T21:56:35.352518+0000","last_clean":"2024-09-04T21:56:35.352518+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:35.352518+0000","last_undegraded":"2024-09-04T21:56:35.352518+0000","last_fullsized":"2024-09-04T21:56:35.352518+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:39:44.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"143'382","reported_seq":3537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653287+0000","last_change":"2024-09-04T21:56:16.490790+0000","last_active":"2024-09-04T21:56:34.653287+0000","last_peered":"2024-09-04T21:56:34.653287+0000","last_clean":"2024-09-04T21:56:34.653287+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:34.653287+0000","last_undegraded":"2024-09-04T21:56:34.653287+0000","last_fullsized":"2024-09-04T21:56:34.653287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:56:20.741571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"143'566","reported_seq":19161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861349+0000","last_change":"2024-09-04T21:56:16.556682+0000","last_active":"2024-09-04T21:56:34.861349+0000","last_peered":"2024-09-04T21:56:34.861349+0000","last_clean":"2024-09-04T21:56:34.861349+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:34.861349+0000","last_undegraded":"2024-09-04T21:56:34.861349+0000","last_fullsized":"2024-09-04T21:56:34.861349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:18:12.735862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"144'394","reported_seq":2484,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182638+0000","last_change":"2024-09-04T21:56:29.616344+0000","last_active":"2024-09-04T21:56:35.182638+0000","last_peered":"2024-09-04T21:56:35.182638+0000","last_clean":"2024-09-04T21:56:35.182638+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:35.182638+0000","last_undegraded":"2024-09-04T21:56:35.182638+0000","last_fullsized":"2024-09-04T21:56:35.182638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:43:32.577364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"143'290","reported_seq":1711,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352394+0000","last_change":"2024-09-04T21:56:16.527857+0000","last_active":"2024-09-04T21:56:35.352394+0000","last_peered":"2024-09-04T21:56:35.352394+0000","last_clean":"2024-09-04T21:56:35.352394+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:35.352394+0000","last_undegraded":"2024-09-04T21:56:35.352394+0000","last_fullsized":"2024-09-04T21:56:35.352394+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:15:56.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"143'957","reported_seq":17593,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652386+0000","last_change":"2024-09-04T21:56:16.488214+0000","last_active":"2024-09-04T21:56:34.652386+0000","last_peered":"2024-09-04T21:56:34.652386+0000","last_clean":"2024-09-04T21:56:34.652386+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:34.652386+0000","last_undegraded":"2024-09-04T21:56:34.652386+0000","last_fullsized":"2024-09-04T21:56:34.652386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:19.822098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"143'1213","reported_seq":3290,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654113+0000","last_change":"2024-09-04T21:56:16.491514+0000","last_active":"2024-09-04T21:56:34.654113+0000","last_peered":"2024-09-04T21:56:34.654113+0000","last_clean":"2024-09-04T21:56:34.654113+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:34.654113+0000","last_undegraded":"2024-09-04T21:56:34.654113+0000","last_fullsized":"2024-09-04T21:56:34.654113+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:21.020845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'402","reported_seq":4769,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653256+0000","last_change":"2024-09-04T21:56:16.488004+0000","last_active":"2024-09-04T21:56:34.653256+0000","last_peered":"2024-09-04T21:56:34.653256+0000","last_clean":"2024-09-04T21:56:34.653256+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:34.653256+0000","last_undegraded":"2024-09-04T21:56:34.653256+0000","last_fullsized":"2024-09-04T21:56:34.653256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:00.780840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"144'956","reported_seq":3245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352481+0000","last_change":"2024-09-04T21:56:16.528237+0000","last_active":"2024-09-04T21:56:35.352481+0000","last_peered":"2024-09-04T21:56:35.352481+0000","last_clean":"2024-09-04T21:56:35.352481+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:35.352481+0000","last_undegraded":"2024-09-04T21:56:35.352481+0000","last_fullsized":"2024-09-04T21:56:35.352481+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:49:16.916644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'357","reported_seq":2708,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.183014+0000","last_change":"2024-09-04T21:56:29.617384+0000","last_active":"2024-09-04T21:56:35.183014+0000","last_peered":"2024-09-04T21:56:35.183014+0000","last_clean":"2024-09-04T21:56:35.183014+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:35.183014+0000","last_undegraded":"2024-09-04T21:56:35.183014+0000","last_fullsized":"2024-09-04T21:56:35.183014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:30:08.502145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"144'393","reported_seq":3169,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227566+0000","last_change":"2024-09-04T21:56:16.600842+0000","last_active":"2024-09-04T21:56:35.227566+0000","last_peered":"2024-09-04T21:56:35.227566+0000","last_clean":"2024-09-04T21:56:35.227566+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:35.227566+0000","last_undegraded":"2024-09-04T21:56:35.227566+0000","last_fullsized":"2024-09-04T21:56:35.227566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:00.160039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"143'345","reported_seq":2075,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652357+0000","last_change":"2024-09-04T21:56:16.488308+0000","last_active":"2024-09-04T21:56:34.652357+0000","last_peered":"2024-09-04T21:56:34.652357+0000","last_clean":"2024-09-04T21:56:34.652357+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:34.652357+0000","last_undegraded":"2024-09-04T21:56:34.652357+0000","last_fullsized":"2024-09-04T21:56:34.652357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:24:16.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353101+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:35.353101+0000","last_peered":"2024-09-04T21:56:35.353101+0000","last_clean":"2024-09-04T21:56:35.353101+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:35.353101+0000","last_undegraded":"2024-09-04T21:56:35.353101+0000","last_fullsized":"2024-09-04T21:56:35.353101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"144'465","reported_seq":2549,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352500+0000","last_change":"2024-09-04T21:56:16.527617+0000","last_active":"2024-09-04T21:56:35.352500+0000","last_peered":"2024-09-04T21:56:35.352500+0000","last_clean":"2024-09-04T21:56:35.352500+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:35.352500+0000","last_undegraded":"2024-09-04T21:56:35.352500+0000","last_fullsized":"2024-09-04T21:56:35.352500+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:34:20.478729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654438+0000","last_change":"2024-09-04T21:55:50.373208+0000","last_active":"2024-09-04T21:56:34.654438+0000","last_peered":"2024-09-04T21:56:34.654438+0000","last_clean":"2024-09-04T21:56:34.654438+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:34.654438+0000","last_undegraded":"2024-09-04T21:56:34.654438+0000","last_fullsized":"2024-09-04T21:56:34.654438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:34.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"143'353","reported_seq":3575,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653090+0000","last_change":"2024-09-04T21:56:16.491647+0000","last_active":"2024-09-04T21:56:34.653090+0000","last_peered":"2024-09-04T21:56:34.653090+0000","last_clean":"2024-09-04T21:56:34.653090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:34.653090+0000","last_undegraded":"2024-09-04T21:56:34.653090+0000","last_fullsized":"2024-09-04T21:56:34.653090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:45:54.303973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227837+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:35.227837+0000","last_peered":"2024-09-04T21:56:35.227837+0000","last_clean":"2024-09-04T21:56:35.227837+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:35.227837+0000","last_undegraded":"2024-09-04T21:56:35.227837+0000","last_fullsized":"2024-09-04T21:56:35.227837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"143'330","reported_seq":1971,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654382+0000","last_change":"2024-09-04T21:56:16.492759+0000","last_active":"2024-09-04T21:56:34.654382+0000","last_peered":"2024-09-04T21:56:34.654382+0000","last_clean":"2024-09-04T21:56:34.654382+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:34.654382+0000","last_undegraded":"2024-09-04T21:56:34.654382+0000","last_fullsized":"2024-09-04T21:56:34.654382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:13:53.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654037+0000","last_change":"2024-09-04T21:56:01.380611+0000","last_active":"2024-09-04T21:56:34.654037+0000","last_peered":"2024-09-04T21:56:34.654037+0000","last_clean":"2024-09-04T21:56:34.654037+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:34.654037+0000","last_undegraded":"2024-09-04T21:56:34.654037+0000","last_fullsized":"2024-09-04T21:56:34.654037+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:44.108771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'560","reported_seq":11918,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182904+0000","last_change":"2024-09-04T21:56:29.614226+0000","last_active":"2024-09-04T21:56:35.182904+0000","last_peered":"2024-09-04T21:56:35.182904+0000","last_clean":"2024-09-04T21:56:35.182904+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:35.182904+0000","last_undegraded":"2024-09-04T21:56:35.182904+0000","last_fullsized":"2024-09-04T21:56:35.182904+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:09:39.718023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"144'509","reported_seq":4121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652418+0000","last_change":"2024-09-04T21:56:16.490969+0000","last_active":"2024-09-04T21:56:34.652418+0000","last_peered":"2024-09-04T21:56:34.652418+0000","last_clean":"2024-09-04T21:56:34.652418+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:34.652418+0000","last_undegraded":"2024-09-04T21:56:34.652418+0000","last_fullsized":"2024-09-04T21:56:34.652418+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:46:12.808214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182664+0000","last_change":"2024-09-04T21:56:29.610468+0000","last_active":"2024-09-04T21:56:35.182664+0000","last_peered":"2024-09-04T21:56:35.182664+0000","last_clean":"2024-09-04T21:56:35.182664+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:35.182664+0000","last_undegraded":"2024-09-04T21:56:35.182664+0000","last_fullsized":"2024-09-04T21:56:35.182664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:27:48.891402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"143'376","reported_seq":3305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653993+0000","last_change":"2024-09-04T21:56:16.493672+0000","last_active":"2024-09-04T21:56:34.653993+0000","last_peered":"2024-09-04T21:56:34.653993+0000","last_clean":"2024-09-04T21:56:34.653993+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:34.653993+0000","last_undegraded":"2024-09-04T21:56:34.653993+0000","last_fullsized":"2024-09-04T21:56:34.653993+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:44:32.875548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861461+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:56:34.861461+0000","last_peered":"2024-09-04T21:56:34.861461+0000","last_clean":"2024-09-04T21:56:34.861461+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:34.861461+0000","last_undegraded":"2024-09-04T21:56:34.861461+0000","last_fullsized":"2024-09-04T21:56:34.861461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'618","reported_seq":4691,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352208+0000","last_change":"2024-09-04T21:56:16.527386+0000","last_active":"2024-09-04T21:56:35.352208+0000","last_peered":"2024-09-04T21:56:35.352208+0000","last_clean":"2024-09-04T21:56:35.352208+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:35.352208+0000","last_undegraded":"2024-09-04T21:56:35.352208+0000","last_fullsized":"2024-09-04T21:56:35.352208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:25:55.616051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227578+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:35.227578+0000","last_peered":"2024-09-04T21:56:35.227578+0000","last_clean":"2024-09-04T21:56:35.227578+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:35.227578+0000","last_undegraded":"2024-09-04T21:56:35.227578+0000","last_fullsized":"2024-09-04T21:56:35.227578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"144'426","reported_seq":3648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652929+0000","last_change":"2024-09-04T21:56:16.487890+0000","last_active":"2024-09-04T21:56:34.652929+0000","last_peered":"2024-09-04T21:56:34.652929+0000","last_clean":"2024-09-04T21:56:34.652929+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:34.652929+0000","last_undegraded":"2024-09-04T21:56:34.652929+0000","last_fullsized":"2024-09-04T21:56:34.652929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:00:48.401698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":115,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182685+0000","last_change":"2024-09-04T21:56:29.613744+0000","last_active":"2024-09-04T21:56:35.182685+0000","last_peered":"2024-09-04T21:56:35.182685+0000","last_clean":"2024-09-04T21:56:35.182685+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:35.182685+0000","last_undegraded":"2024-09-04T21:56:35.182685+0000","last_fullsized":"2024-09-04T21:56:35.182685+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:26.686223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227261+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:35.227261+0000","last_peered":"2024-09-04T21:56:35.227261+0000","last_clean":"2024-09-04T21:56:35.227261+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:35.227261+0000","last_undegraded":"2024-09-04T21:56:35.227261+0000","last_fullsized":"2024-09-04T21:56:35.227261+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"144'590","reported_seq":3820,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652932+0000","last_change":"2024-09-04T21:56:16.489907+0000","last_active":"2024-09-04T21:56:34.652932+0000","last_peered":"2024-09-04T21:56:34.652932+0000","last_clean":"2024-09-04T21:56:34.652932+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:34.652932+0000","last_undegraded":"2024-09-04T21:56:34.652932+0000","last_fullsized":"2024-09-04T21:56:34.652932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:10:28.643680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"143'345","reported_seq":2203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652276+0000","last_change":"2024-09-04T21:56:16.491283+0000","last_active":"2024-09-04T21:56:34.652276+0000","last_peered":"2024-09-04T21:56:34.652276+0000","last_clean":"2024-09-04T21:56:34.652276+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:34.652276+0000","last_undegraded":"2024-09-04T21:56:34.652276+0000","last_fullsized":"2024-09-04T21:56:34.652276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:43:39.964868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"143'706","reported_seq":2977,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353002+0000","last_change":"2024-09-04T21:56:16.584413+0000","last_active":"2024-09-04T21:56:35.353002+0000","last_peered":"2024-09-04T21:56:35.353002+0000","last_clean":"2024-09-04T21:56:35.353002+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:35.353002+0000","last_undegraded":"2024-09-04T21:56:35.353002+0000","last_fullsized":"2024-09-04T21:56:35.353002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:01:21.655483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"143'383","reported_seq":2457,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653129+0000","last_change":"2024-09-04T21:56:16.488947+0000","last_active":"2024-09-04T21:56:34.653129+0000","last_peered":"2024-09-04T21:56:34.653129+0000","last_clean":"2024-09-04T21:56:34.653129+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:34.653129+0000","last_undegraded":"2024-09-04T21:56:34.653129+0000","last_fullsized":"2024-09-04T21:56:34.653129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:49.649633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"144'357","reported_seq":3597,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182710+0000","last_change":"2024-09-04T21:56:29.613277+0000","last_active":"2024-09-04T21:56:35.182710+0000","last_peered":"2024-09-04T21:56:35.182710+0000","last_clean":"2024-09-04T21:56:35.182710+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:35.182710+0000","last_undegraded":"2024-09-04T21:56:35.182710+0000","last_fullsized":"2024-09-04T21:56:35.182710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:00.762632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'419","reported_seq":11217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227512+0000","last_change":"2024-09-04T21:56:16.600636+0000","last_active":"2024-09-04T21:56:35.227512+0000","last_peered":"2024-09-04T21:56:35.227512+0000","last_clean":"2024-09-04T21:56:35.227512+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:35.227512+0000","last_undegraded":"2024-09-04T21:56:35.227512+0000","last_fullsized":"2024-09-04T21:56:35.227512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:53:20.412017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"143'394","reported_seq":1388,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227770+0000","last_change":"2024-09-04T21:56:16.601950+0000","last_active":"2024-09-04T21:56:35.227770+0000","last_peered":"2024-09-04T21:56:35.227770+0000","last_clean":"2024-09-04T21:56:35.227770+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:35.227770+0000","last_undegraded":"2024-09-04T21:56:35.227770+0000","last_fullsized":"2024-09-04T21:56:35.227770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:42.603261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"143'488","reported_seq":11950,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227095+0000","last_change":"2024-09-04T21:56:16.602325+0000","last_active":"2024-09-04T21:56:35.227095+0000","last_peered":"2024-09-04T21:56:35.227095+0000","last_clean":"2024-09-04T21:56:35.227095+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:35.227095+0000","last_undegraded":"2024-09-04T21:56:35.227095+0000","last_fullsized":"2024-09-04T21:56:35.227095+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:41:05.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"143'830","reported_seq":3709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352971+0000","last_change":"2024-09-04T21:56:16.584288+0000","last_active":"2024-09-04T21:56:35.352971+0000","last_peered":"2024-09-04T21:56:35.352971+0000","last_clean":"2024-09-04T21:56:35.352971+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:35.352971+0000","last_undegraded":"2024-09-04T21:56:35.352971+0000","last_fullsized":"2024-09-04T21:56:35.352971+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:03:59.014910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"144'579","reported_seq":6064,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227102+0000","last_change":"2024-09-04T21:56:16.598876+0000","last_active":"2024-09-04T21:56:35.227102+0000","last_peered":"2024-09-04T21:56:35.227102+0000","last_clean":"2024-09-04T21:56:35.227102+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:35.227102+0000","last_undegraded":"2024-09-04T21:56:35.227102+0000","last_fullsized":"2024-09-04T21:56:35.227102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:34:16.388010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"143'763","reported_seq":13635,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352207+0000","last_change":"2024-09-04T21:56:16.528688+0000","last_active":"2024-09-04T21:56:35.352207+0000","last_peered":"2024-09-04T21:56:35.352207+0000","last_clean":"2024-09-04T21:56:35.352207+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:35.352207+0000","last_undegraded":"2024-09-04T21:56:35.352207+0000","last_fullsized":"2024-09-04T21:56:35.352207+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:08.477339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"142'973","reported_seq":2644,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227285+0000","last_change":"2024-09-04T21:56:16.596597+0000","last_active":"2024-09-04T21:56:35.227285+0000","last_peered":"2024-09-04T21:56:35.227285+0000","last_clean":"2024-09-04T21:56:35.227285+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:35.227285+0000","last_undegraded":"2024-09-04T21:56:35.227285+0000","last_fullsized":"2024-09-04T21:56:35.227285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:19:55.348359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17483,"ondisk_log_size":17483,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566093,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39276,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332564,"statfs":{"total":96636764160,"available":96596545536,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1945600,"data_stored":1504066,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542183,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45532,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326308,"statfs":{"total":96636764160,"available":96590139392,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34072,"kb_used_data":1916,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337768,"statfs":{"total":96636764160,"available":96601874432,"internally_reserved":0,"allocated":1961984,"data_stored":1504066,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:38.409 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-04T21:56:38.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-04T21:56:38.729 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-04T21:56:38.737 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-04T21:56:38.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:39.020 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:39.021 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:39.041 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-04T21:56:38.472889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355036,"kb_used_data":15052,"kb_used_omap":46,"kb_used_meta":339729,"kb_avail":754619684,"statfs":{"total":773094113280,"available":772730556416,"internally_reserved":0,"allocated":15413248,"data_stored":8519606,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":347882642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001923"},"pg_stats":[{"pgid":"3.1f","version":"144'764","reported_seq":2993,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352518+0000","last_change":"2024-09-04T21:56:16.529040+0000","last_active":"2024-09-04T21:56:35.352518+0000","last_peered":"2024-09-04T21:56:35.352518+0000","last_clean":"2024-09-04T21:56:35.352518+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:35.352518+0000","last_undegraded":"2024-09-04T21:56:35.352518+0000","last_fullsized":"2024-09-04T21:56:35.352518+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:39:44.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"143'382","reported_seq":3537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653287+0000","last_change":"2024-09-04T21:56:16.490790+0000","last_active":"2024-09-04T21:56:34.653287+0000","last_peered":"2024-09-04T21:56:34.653287+0000","last_clean":"2024-09-04T21:56:34.653287+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:34.653287+0000","last_undegraded":"2024-09-04T21:56:34.653287+0000","last_fullsized":"2024-09-04T21:56:34.653287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:56:20.741571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"143'566","reported_seq":19161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861349+0000","last_change":"2024-09-04T21:56:16.556682+0000","last_active":"2024-09-04T21:56:34.861349+0000","last_peered":"2024-09-04T21:56:34.861349+0000","last_clean":"2024-09-04T21:56:34.861349+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:34.861349+0000","last_undegraded":"2024-09-04T21:56:34.861349+0000","last_fullsized":"2024-09-04T21:56:34.861349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:18:12.735862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"144'394","reported_seq":2484,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182638+0000","last_change":"2024-09-04T21:56:29.616344+0000","last_active":"2024-09-04T21:56:35.182638+0000","last_peered":"2024-09-04T21:56:35.182638+0000","last_clean":"2024-09-04T21:56:35.182638+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:35.182638+0000","last_undegraded":"2024-09-04T21:56:35.182638+0000","last_fullsized":"2024-09-04T21:56:35.182638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:43:32.577364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"143'290","reported_seq":1711,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352394+0000","last_change":"2024-09-04T21:56:16.527857+0000","last_active":"2024-09-04T21:56:35.352394+0000","last_peered":"2024-09-04T21:56:35.352394+0000","last_clean":"2024-09-04T21:56:35.352394+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:35.352394+0000","last_undegraded":"2024-09-04T21:56:35.352394+0000","last_fullsized":"2024-09-04T21:56:35.352394+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:15:56.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"143'957","reported_seq":17593,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652386+0000","last_change":"2024-09-04T21:56:16.488214+0000","last_active":"2024-09-04T21:56:34.652386+0000","last_peered":"2024-09-04T21:56:34.652386+0000","last_clean":"2024-09-04T21:56:34.652386+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:34.652386+0000","last_undegraded":"2024-09-04T21:56:34.652386+0000","last_fullsized":"2024-09-04T21:56:34.652386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:19.822098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"143'1213","reported_seq":3290,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654113+0000","last_change":"2024-09-04T21:56:16.491514+0000","last_active":"2024-09-04T21:56:34.654113+0000","last_peered":"2024-09-04T21:56:34.654113+0000","last_clean":"2024-09-04T21:56:34.654113+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:34.654113+0000","last_undegraded":"2024-09-04T21:56:34.654113+0000","last_fullsized":"2024-09-04T21:56:34.654113+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:21.020845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'402","reported_seq":4769,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653256+0000","last_change":"2024-09-04T21:56:16.488004+0000","last_active":"2024-09-04T21:56:34.653256+0000","last_peered":"2024-09-04T21:56:34.653256+0000","last_clean":"2024-09-04T21:56:34.653256+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:34.653256+0000","last_undegraded":"2024-09-04T21:56:34.653256+0000","last_fullsized":"2024-09-04T21:56:34.653256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:00.780840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"144'956","reported_seq":3245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352481+0000","last_change":"2024-09-04T21:56:16.528237+0000","last_active":"2024-09-04T21:56:35.352481+0000","last_peered":"2024-09-04T21:56:35.352481+0000","last_clean":"2024-09-04T21:56:35.352481+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:35.352481+0000","last_undegraded":"2024-09-04T21:56:35.352481+0000","last_fullsized":"2024-09-04T21:56:35.352481+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:49:16.916644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'357","reported_seq":2708,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.183014+0000","last_change":"2024-09-04T21:56:29.617384+0000","last_active":"2024-09-04T21:56:35.183014+0000","last_peered":"2024-09-04T21:56:35.183014+0000","last_clean":"2024-09-04T21:56:35.183014+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:35.183014+0000","last_undegraded":"2024-09-04T21:56:35.183014+0000","last_fullsized":"2024-09-04T21:56:35.183014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:30:08.502145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"144'393","reported_seq":3169,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227566+0000","last_change":"2024-09-04T21:56:16.600842+0000","last_active":"2024-09-04T21:56:35.227566+0000","last_peered":"2024-09-04T21:56:35.227566+0000","last_clean":"2024-09-04T21:56:35.227566+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:35.227566+0000","last_undegraded":"2024-09-04T21:56:35.227566+0000","last_fullsized":"2024-09-04T21:56:35.227566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:00.160039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"143'345","reported_seq":2075,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652357+0000","last_change":"2024-09-04T21:56:16.488308+0000","last_active":"2024-09-04T21:56:34.652357+0000","last_peered":"2024-09-04T21:56:34.652357+0000","last_clean":"2024-09-04T21:56:34.652357+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:34.652357+0000","last_undegraded":"2024-09-04T21:56:34.652357+0000","last_fullsized":"2024-09-04T21:56:34.652357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:24:16.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353101+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:35.353101+0000","last_peered":"2024-09-04T21:56:35.353101+0000","last_clean":"2024-09-04T21:56:35.353101+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:35.353101+0000","last_undegraded":"2024-09-04T21:56:35.353101+0000","last_fullsized":"2024-09-04T21:56:35.353101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"144'465","reported_seq":2549,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352500+0000","last_change":"2024-09-04T21:56:16.527617+0000","last_active":"2024-09-04T21:56:35.352500+0000","last_peered":"2024-09-04T21:56:35.352500+0000","last_clean":"2024-09-04T21:56:35.352500+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:35.352500+0000","last_undegraded":"2024-09-04T21:56:35.352500+0000","last_fullsized":"2024-09-04T21:56:35.352500+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:34:20.478729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654438+0000","last_change":"2024-09-04T21:55:50.373208+0000","last_active":"2024-09-04T21:56:34.654438+0000","last_peered":"2024-09-04T21:56:34.654438+0000","last_clean":"2024-09-04T21:56:34.654438+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:34.654438+0000","last_undegraded":"2024-09-04T21:56:34.654438+0000","last_fullsized":"2024-09-04T21:56:34.654438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:34.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"143'353","reported_seq":3575,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653090+0000","last_change":"2024-09-04T21:56:16.491647+0000","last_active":"2024-09-04T21:56:34.653090+0000","last_peered":"2024-09-04T21:56:34.653090+0000","last_clean":"2024-09-04T21:56:34.653090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:34.653090+0000","last_undegraded":"2024-09-04T21:56:34.653090+0000","last_fullsized":"2024-09-04T21:56:34.653090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:45:54.303973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227837+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:35.227837+0000","last_peered":"2024-09-04T21:56:35.227837+0000","last_clean":"2024-09-04T21:56:35.227837+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:35.227837+0000","last_undegraded":"2024-09-04T21:56:35.227837+0000","last_fullsized":"2024-09-04T21:56:35.227837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"143'330","reported_seq":1971,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654382+0000","last_change":"2024-09-04T21:56:16.492759+0000","last_active":"2024-09-04T21:56:34.654382+0000","last_peered":"2024-09-04T21:56:34.654382+0000","last_clean":"2024-09-04T21:56:34.654382+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:34.654382+0000","last_undegraded":"2024-09-04T21:56:34.654382+0000","last_fullsized":"2024-09-04T21:56:34.654382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:13:53.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654037+0000","last_change":"2024-09-04T21:56:01.380611+0000","last_active":"2024-09-04T21:56:34.654037+0000","last_peered":"2024-09-04T21:56:34.654037+0000","last_clean":"2024-09-04T21:56:34.654037+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:34.654037+0000","last_undegraded":"2024-09-04T21:56:34.654037+0000","last_fullsized":"2024-09-04T21:56:34.654037+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:44.108771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'560","reported_seq":11918,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182904+0000","last_change":"2024-09-04T21:56:29.614226+0000","last_active":"2024-09-04T21:56:35.182904+0000","last_peered":"2024-09-04T21:56:35.182904+0000","last_clean":"2024-09-04T21:56:35.182904+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:35.182904+0000","last_undegraded":"2024-09-04T21:56:35.182904+0000","last_fullsized":"2024-09-04T21:56:35.182904+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:09:39.718023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"144'509","reported_seq":4121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652418+0000","last_change":"2024-09-04T21:56:16.490969+0000","last_active":"2024-09-04T21:56:34.652418+0000","last_peered":"2024-09-04T21:56:34.652418+0000","last_clean":"2024-09-04T21:56:34.652418+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:34.652418+0000","last_undegraded":"2024-09-04T21:56:34.652418+0000","last_fullsized":"2024-09-04T21:56:34.652418+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:46:12.808214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182664+0000","last_change":"2024-09-04T21:56:29.610468+0000","last_active":"2024-09-04T21:56:35.182664+0000","last_peered":"2024-09-04T21:56:35.182664+0000","last_clean":"2024-09-04T21:56:35.182664+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:35.182664+0000","last_undegraded":"2024-09-04T21:56:35.182664+0000","last_fullsized":"2024-09-04T21:56:35.182664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:27:48.891402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"143'376","reported_seq":3305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653993+0000","last_change":"2024-09-04T21:56:16.493672+0000","last_active":"2024-09-04T21:56:34.653993+0000","last_peered":"2024-09-04T21:56:34.653993+0000","last_clean":"2024-09-04T21:56:34.653993+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:34.653993+0000","last_undegraded":"2024-09-04T21:56:34.653993+0000","last_fullsized":"2024-09-04T21:56:34.653993+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:44:32.875548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861461+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:56:34.861461+0000","last_peered":"2024-09-04T21:56:34.861461+0000","last_clean":"2024-09-04T21:56:34.861461+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:34.861461+0000","last_undegraded":"2024-09-04T21:56:34.861461+0000","last_fullsized":"2024-09-04T21:56:34.861461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'618","reported_seq":4691,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352208+0000","last_change":"2024-09-04T21:56:16.527386+0000","last_active":"2024-09-04T21:56:35.352208+0000","last_peered":"2024-09-04T21:56:35.352208+0000","last_clean":"2024-09-04T21:56:35.352208+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:35.352208+0000","last_undegraded":"2024-09-04T21:56:35.352208+0000","last_fullsized":"2024-09-04T21:56:35.352208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:25:55.616051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227578+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:35.227578+0000","last_peered":"2024-09-04T21:56:35.227578+0000","last_clean":"2024-09-04T21:56:35.227578+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:35.227578+0000","last_undegraded":"2024-09-04T21:56:35.227578+0000","last_fullsized":"2024-09-04T21:56:35.227578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"144'426","reported_seq":3648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652929+0000","last_change":"2024-09-04T21:56:16.487890+0000","last_active":"2024-09-04T21:56:34.652929+0000","last_peered":"2024-09-04T21:56:34.652929+0000","last_clean":"2024-09-04T21:56:34.652929+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:34.652929+0000","last_undegraded":"2024-09-04T21:56:34.652929+0000","last_fullsized":"2024-09-04T21:56:34.652929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:00:48.401698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":115,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182685+0000","last_change":"2024-09-04T21:56:29.613744+0000","last_active":"2024-09-04T21:56:35.182685+0000","last_peered":"2024-09-04T21:56:35.182685+0000","last_clean":"2024-09-04T21:56:35.182685+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:35.182685+0000","last_undegraded":"2024-09-04T21:56:35.182685+0000","last_fullsized":"2024-09-04T21:56:35.182685+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:26.686223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227261+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:35.227261+0000","last_peered":"2024-09-04T21:56:35.227261+0000","last_clean":"2024-09-04T21:56:35.227261+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:35.227261+0000","last_undegraded":"2024-09-04T21:56:35.227261+0000","last_fullsized":"2024-09-04T21:56:35.227261+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"144'590","reported_seq":3820,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652932+0000","last_change":"2024-09-04T21:56:16.489907+0000","last_active":"2024-09-04T21:56:34.652932+0000","last_peered":"2024-09-04T21:56:34.652932+0000","last_clean":"2024-09-04T21:56:34.652932+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:34.652932+0000","last_undegraded":"2024-09-04T21:56:34.652932+0000","last_fullsized":"2024-09-04T21:56:34.652932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:10:28.643680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"143'345","reported_seq":2203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652276+0000","last_change":"2024-09-04T21:56:16.491283+0000","last_active":"2024-09-04T21:56:34.652276+0000","last_peered":"2024-09-04T21:56:34.652276+0000","last_clean":"2024-09-04T21:56:34.652276+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:34.652276+0000","last_undegraded":"2024-09-04T21:56:34.652276+0000","last_fullsized":"2024-09-04T21:56:34.652276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:43:39.964868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"143'706","reported_seq":2977,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353002+0000","last_change":"2024-09-04T21:56:16.584413+0000","last_active":"2024-09-04T21:56:35.353002+0000","last_peered":"2024-09-04T21:56:35.353002+0000","last_clean":"2024-09-04T21:56:35.353002+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:35.353002+0000","last_undegraded":"2024-09-04T21:56:35.353002+0000","last_fullsized":"2024-09-04T21:56:35.353002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:01:21.655483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"143'383","reported_seq":2457,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653129+0000","last_change":"2024-09-04T21:56:16.488947+0000","last_active":"2024-09-04T21:56:34.653129+0000","last_peered":"2024-09-04T21:56:34.653129+0000","last_clean":"2024-09-04T21:56:34.653129+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:34.653129+0000","last_undegraded":"2024-09-04T21:56:34.653129+0000","last_fullsized":"2024-09-04T21:56:34.653129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:49.649633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"144'357","reported_seq":3597,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182710+0000","last_change":"2024-09-04T21:56:29.613277+0000","last_active":"2024-09-04T21:56:35.182710+0000","last_peered":"2024-09-04T21:56:35.182710+0000","last_clean":"2024-09-04T21:56:35.182710+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:35.182710+0000","last_undegraded":"2024-09-04T21:56:35.182710+0000","last_fullsized":"2024-09-04T21:56:35.182710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:00.762632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'419","reported_seq":11217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227512+0000","last_change":"2024-09-04T21:56:16.600636+0000","last_active":"2024-09-04T21:56:35.227512+0000","last_peered":"2024-09-04T21:56:35.227512+0000","last_clean":"2024-09-04T21:56:35.227512+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:35.227512+0000","last_undegraded":"2024-09-04T21:56:35.227512+0000","last_fullsized":"2024-09-04T21:56:35.227512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:53:20.412017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"143'394","reported_seq":1388,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227770+0000","last_change":"2024-09-04T21:56:16.601950+0000","last_active":"2024-09-04T21:56:35.227770+0000","last_peered":"2024-09-04T21:56:35.227770+0000","last_clean":"2024-09-04T21:56:35.227770+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:35.227770+0000","last_undegraded":"2024-09-04T21:56:35.227770+0000","last_fullsized":"2024-09-04T21:56:35.227770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:42.603261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"143'488","reported_seq":11950,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227095+0000","last_change":"2024-09-04T21:56:16.602325+0000","last_active":"2024-09-04T21:56:35.227095+0000","last_peered":"2024-09-04T21:56:35.227095+0000","last_clean":"2024-09-04T21:56:35.227095+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:35.227095+0000","last_undegraded":"2024-09-04T21:56:35.227095+0000","last_fullsized":"2024-09-04T21:56:35.227095+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:41:05.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"143'830","reported_seq":3709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352971+0000","last_change":"2024-09-04T21:56:16.584288+0000","last_active":"2024-09-04T21:56:35.352971+0000","last_peered":"2024-09-04T21:56:35.352971+0000","last_clean":"2024-09-04T21:56:35.352971+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:35.352971+0000","last_undegraded":"2024-09-04T21:56:35.352971+0000","last_fullsized":"2024-09-04T21:56:35.352971+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:03:59.014910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"144'579","reported_seq":6064,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227102+0000","last_change":"2024-09-04T21:56:16.598876+0000","last_active":"2024-09-04T21:56:35.227102+0000","last_peered":"2024-09-04T21:56:35.227102+0000","last_clean":"2024-09-04T21:56:35.227102+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:35.227102+0000","last_undegraded":"2024-09-04T21:56:35.227102+0000","last_fullsized":"2024-09-04T21:56:35.227102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:34:16.388010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"143'763","reported_seq":13635,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352207+0000","last_change":"2024-09-04T21:56:16.528688+0000","last_active":"2024-09-04T21:56:35.352207+0000","last_peered":"2024-09-04T21:56:35.352207+0000","last_clean":"2024-09-04T21:56:35.352207+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:35.352207+0000","last_undegraded":"2024-09-04T21:56:35.352207+0000","last_fullsized":"2024-09-04T21:56:35.352207+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:08.477339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"142'973","reported_seq":2644,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227285+0000","last_change":"2024-09-04T21:56:16.596597+0000","last_active":"2024-09-04T21:56:35.227285+0000","last_peered":"2024-09-04T21:56:35.227285+0000","last_clean":"2024-09-04T21:56:35.227285+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:35.227285+0000","last_undegraded":"2024-09-04T21:56:35.227285+0000","last_fullsized":"2024-09-04T21:56:35.227285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:19:55.348359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17483,"ondisk_log_size":17483,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566094,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39276,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332564,"statfs":{"total":96636764160,"available":96596545536,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1945600,"data_stored":1504066,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45548,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326292,"statfs":{"total":96636764160,"available":96590123008,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34072,"kb_used_data":1916,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337768,"statfs":{"total":96636764160,"available":96601874432,"internally_reserved":0,"allocated":1961984,"data_stored":1504066,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:39.043 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:39.340 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:39.341 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:39.356 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-04T21:56:38.472889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355036,"kb_used_data":15052,"kb_used_omap":46,"kb_used_meta":339729,"kb_avail":754619684,"statfs":{"total":773094113280,"available":772730556416,"internally_reserved":0,"allocated":15413248,"data_stored":8519606,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":347882642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001923"},"pg_stats":[{"pgid":"3.1f","version":"144'764","reported_seq":2993,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352518+0000","last_change":"2024-09-04T21:56:16.529040+0000","last_active":"2024-09-04T21:56:35.352518+0000","last_peered":"2024-09-04T21:56:35.352518+0000","last_clean":"2024-09-04T21:56:35.352518+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:35.352518+0000","last_undegraded":"2024-09-04T21:56:35.352518+0000","last_fullsized":"2024-09-04T21:56:35.352518+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:39:44.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"143'382","reported_seq":3537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653287+0000","last_change":"2024-09-04T21:56:16.490790+0000","last_active":"2024-09-04T21:56:34.653287+0000","last_peered":"2024-09-04T21:56:34.653287+0000","last_clean":"2024-09-04T21:56:34.653287+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:34.653287+0000","last_undegraded":"2024-09-04T21:56:34.653287+0000","last_fullsized":"2024-09-04T21:56:34.653287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:56:20.741571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"143'566","reported_seq":19161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861349+0000","last_change":"2024-09-04T21:56:16.556682+0000","last_active":"2024-09-04T21:56:34.861349+0000","last_peered":"2024-09-04T21:56:34.861349+0000","last_clean":"2024-09-04T21:56:34.861349+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:34.861349+0000","last_undegraded":"2024-09-04T21:56:34.861349+0000","last_fullsized":"2024-09-04T21:56:34.861349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:18:12.735862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"144'394","reported_seq":2484,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182638+0000","last_change":"2024-09-04T21:56:29.616344+0000","last_active":"2024-09-04T21:56:35.182638+0000","last_peered":"2024-09-04T21:56:35.182638+0000","last_clean":"2024-09-04T21:56:35.182638+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:35.182638+0000","last_undegraded":"2024-09-04T21:56:35.182638+0000","last_fullsized":"2024-09-04T21:56:35.182638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:43:32.577364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"143'290","reported_seq":1711,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352394+0000","last_change":"2024-09-04T21:56:16.527857+0000","last_active":"2024-09-04T21:56:35.352394+0000","last_peered":"2024-09-04T21:56:35.352394+0000","last_clean":"2024-09-04T21:56:35.352394+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:35.352394+0000","last_undegraded":"2024-09-04T21:56:35.352394+0000","last_fullsized":"2024-09-04T21:56:35.352394+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:15:56.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"143'957","reported_seq":17593,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652386+0000","last_change":"2024-09-04T21:56:16.488214+0000","last_active":"2024-09-04T21:56:34.652386+0000","last_peered":"2024-09-04T21:56:34.652386+0000","last_clean":"2024-09-04T21:56:34.652386+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:34.652386+0000","last_undegraded":"2024-09-04T21:56:34.652386+0000","last_fullsized":"2024-09-04T21:56:34.652386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:19.822098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"143'1213","reported_seq":3290,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654113+0000","last_change":"2024-09-04T21:56:16.491514+0000","last_active":"2024-09-04T21:56:34.654113+0000","last_peered":"2024-09-04T21:56:34.654113+0000","last_clean":"2024-09-04T21:56:34.654113+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:34.654113+0000","last_undegraded":"2024-09-04T21:56:34.654113+0000","last_fullsized":"2024-09-04T21:56:34.654113+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:21.020845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'402","reported_seq":4769,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653256+0000","last_change":"2024-09-04T21:56:16.488004+0000","last_active":"2024-09-04T21:56:34.653256+0000","last_peered":"2024-09-04T21:56:34.653256+0000","last_clean":"2024-09-04T21:56:34.653256+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:34.653256+0000","last_undegraded":"2024-09-04T21:56:34.653256+0000","last_fullsized":"2024-09-04T21:56:34.653256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:00.780840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"144'956","reported_seq":3245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352481+0000","last_change":"2024-09-04T21:56:16.528237+0000","last_active":"2024-09-04T21:56:35.352481+0000","last_peered":"2024-09-04T21:56:35.352481+0000","last_clean":"2024-09-04T21:56:35.352481+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:35.352481+0000","last_undegraded":"2024-09-04T21:56:35.352481+0000","last_fullsized":"2024-09-04T21:56:35.352481+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:49:16.916644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'357","reported_seq":2708,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.183014+0000","last_change":"2024-09-04T21:56:29.617384+0000","last_active":"2024-09-04T21:56:35.183014+0000","last_peered":"2024-09-04T21:56:35.183014+0000","last_clean":"2024-09-04T21:56:35.183014+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:35.183014+0000","last_undegraded":"2024-09-04T21:56:35.183014+0000","last_fullsized":"2024-09-04T21:56:35.183014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:30:08.502145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"144'393","reported_seq":3169,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227566+0000","last_change":"2024-09-04T21:56:16.600842+0000","last_active":"2024-09-04T21:56:35.227566+0000","last_peered":"2024-09-04T21:56:35.227566+0000","last_clean":"2024-09-04T21:56:35.227566+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:35.227566+0000","last_undegraded":"2024-09-04T21:56:35.227566+0000","last_fullsized":"2024-09-04T21:56:35.227566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:00.160039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"143'345","reported_seq":2075,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652357+0000","last_change":"2024-09-04T21:56:16.488308+0000","last_active":"2024-09-04T21:56:34.652357+0000","last_peered":"2024-09-04T21:56:34.652357+0000","last_clean":"2024-09-04T21:56:34.652357+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:34.652357+0000","last_undegraded":"2024-09-04T21:56:34.652357+0000","last_fullsized":"2024-09-04T21:56:34.652357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:24:16.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353101+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:35.353101+0000","last_peered":"2024-09-04T21:56:35.353101+0000","last_clean":"2024-09-04T21:56:35.353101+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:35.353101+0000","last_undegraded":"2024-09-04T21:56:35.353101+0000","last_fullsized":"2024-09-04T21:56:35.353101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"144'465","reported_seq":2549,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352500+0000","last_change":"2024-09-04T21:56:16.527617+0000","last_active":"2024-09-04T21:56:35.352500+0000","last_peered":"2024-09-04T21:56:35.352500+0000","last_clean":"2024-09-04T21:56:35.352500+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:35.352500+0000","last_undegraded":"2024-09-04T21:56:35.352500+0000","last_fullsized":"2024-09-04T21:56:35.352500+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:34:20.478729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654438+0000","last_change":"2024-09-04T21:55:50.373208+0000","last_active":"2024-09-04T21:56:34.654438+0000","last_peered":"2024-09-04T21:56:34.654438+0000","last_clean":"2024-09-04T21:56:34.654438+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:34.654438+0000","last_undegraded":"2024-09-04T21:56:34.654438+0000","last_fullsized":"2024-09-04T21:56:34.654438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:34.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"143'353","reported_seq":3575,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653090+0000","last_change":"2024-09-04T21:56:16.491647+0000","last_active":"2024-09-04T21:56:34.653090+0000","last_peered":"2024-09-04T21:56:34.653090+0000","last_clean":"2024-09-04T21:56:34.653090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:34.653090+0000","last_undegraded":"2024-09-04T21:56:34.653090+0000","last_fullsized":"2024-09-04T21:56:34.653090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:45:54.303973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227837+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:35.227837+0000","last_peered":"2024-09-04T21:56:35.227837+0000","last_clean":"2024-09-04T21:56:35.227837+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:35.227837+0000","last_undegraded":"2024-09-04T21:56:35.227837+0000","last_fullsized":"2024-09-04T21:56:35.227837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"143'330","reported_seq":1971,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654382+0000","last_change":"2024-09-04T21:56:16.492759+0000","last_active":"2024-09-04T21:56:34.654382+0000","last_peered":"2024-09-04T21:56:34.654382+0000","last_clean":"2024-09-04T21:56:34.654382+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:34.654382+0000","last_undegraded":"2024-09-04T21:56:34.654382+0000","last_fullsized":"2024-09-04T21:56:34.654382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:13:53.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654037+0000","last_change":"2024-09-04T21:56:01.380611+0000","last_active":"2024-09-04T21:56:34.654037+0000","last_peered":"2024-09-04T21:56:34.654037+0000","last_clean":"2024-09-04T21:56:34.654037+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:34.654037+0000","last_undegraded":"2024-09-04T21:56:34.654037+0000","last_fullsized":"2024-09-04T21:56:34.654037+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:44.108771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'560","reported_seq":11918,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182904+0000","last_change":"2024-09-04T21:56:29.614226+0000","last_active":"2024-09-04T21:56:35.182904+0000","last_peered":"2024-09-04T21:56:35.182904+0000","last_clean":"2024-09-04T21:56:35.182904+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:35.182904+0000","last_undegraded":"2024-09-04T21:56:35.182904+0000","last_fullsized":"2024-09-04T21:56:35.182904+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:09:39.718023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"144'509","reported_seq":4121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652418+0000","last_change":"2024-09-04T21:56:16.490969+0000","last_active":"2024-09-04T21:56:34.652418+0000","last_peered":"2024-09-04T21:56:34.652418+0000","last_clean":"2024-09-04T21:56:34.652418+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:34.652418+0000","last_undegraded":"2024-09-04T21:56:34.652418+0000","last_fullsized":"2024-09-04T21:56:34.652418+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:46:12.808214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182664+0000","last_change":"2024-09-04T21:56:29.610468+0000","last_active":"2024-09-04T21:56:35.182664+0000","last_peered":"2024-09-04T21:56:35.182664+0000","last_clean":"2024-09-04T21:56:35.182664+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:35.182664+0000","last_undegraded":"2024-09-04T21:56:35.182664+0000","last_fullsized":"2024-09-04T21:56:35.182664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:27:48.891402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"143'376","reported_seq":3305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653993+0000","last_change":"2024-09-04T21:56:16.493672+0000","last_active":"2024-09-04T21:56:34.653993+0000","last_peered":"2024-09-04T21:56:34.653993+0000","last_clean":"2024-09-04T21:56:34.653993+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:34.653993+0000","last_undegraded":"2024-09-04T21:56:34.653993+0000","last_fullsized":"2024-09-04T21:56:34.653993+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:44:32.875548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861461+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:56:34.861461+0000","last_peered":"2024-09-04T21:56:34.861461+0000","last_clean":"2024-09-04T21:56:34.861461+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:34.861461+0000","last_undegraded":"2024-09-04T21:56:34.861461+0000","last_fullsized":"2024-09-04T21:56:34.861461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'618","reported_seq":4691,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352208+0000","last_change":"2024-09-04T21:56:16.527386+0000","last_active":"2024-09-04T21:56:35.352208+0000","last_peered":"2024-09-04T21:56:35.352208+0000","last_clean":"2024-09-04T21:56:35.352208+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:35.352208+0000","last_undegraded":"2024-09-04T21:56:35.352208+0000","last_fullsized":"2024-09-04T21:56:35.352208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:25:55.616051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227578+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:35.227578+0000","last_peered":"2024-09-04T21:56:35.227578+0000","last_clean":"2024-09-04T21:56:35.227578+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:35.227578+0000","last_undegraded":"2024-09-04T21:56:35.227578+0000","last_fullsized":"2024-09-04T21:56:35.227578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"144'426","reported_seq":3648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652929+0000","last_change":"2024-09-04T21:56:16.487890+0000","last_active":"2024-09-04T21:56:34.652929+0000","last_peered":"2024-09-04T21:56:34.652929+0000","last_clean":"2024-09-04T21:56:34.652929+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:34.652929+0000","last_undegraded":"2024-09-04T21:56:34.652929+0000","last_fullsized":"2024-09-04T21:56:34.652929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:00:48.401698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":115,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182685+0000","last_change":"2024-09-04T21:56:29.613744+0000","last_active":"2024-09-04T21:56:35.182685+0000","last_peered":"2024-09-04T21:56:35.182685+0000","last_clean":"2024-09-04T21:56:35.182685+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:35.182685+0000","last_undegraded":"2024-09-04T21:56:35.182685+0000","last_fullsized":"2024-09-04T21:56:35.182685+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:26.686223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227261+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:35.227261+0000","last_peered":"2024-09-04T21:56:35.227261+0000","last_clean":"2024-09-04T21:56:35.227261+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:35.227261+0000","last_undegraded":"2024-09-04T21:56:35.227261+0000","last_fullsized":"2024-09-04T21:56:35.227261+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"144'590","reported_seq":3820,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652932+0000","last_change":"2024-09-04T21:56:16.489907+0000","last_active":"2024-09-04T21:56:34.652932+0000","last_peered":"2024-09-04T21:56:34.652932+0000","last_clean":"2024-09-04T21:56:34.652932+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:34.652932+0000","last_undegraded":"2024-09-04T21:56:34.652932+0000","last_fullsized":"2024-09-04T21:56:34.652932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:10:28.643680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"143'345","reported_seq":2203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652276+0000","last_change":"2024-09-04T21:56:16.491283+0000","last_active":"2024-09-04T21:56:34.652276+0000","last_peered":"2024-09-04T21:56:34.652276+0000","last_clean":"2024-09-04T21:56:34.652276+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:34.652276+0000","last_undegraded":"2024-09-04T21:56:34.652276+0000","last_fullsized":"2024-09-04T21:56:34.652276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:43:39.964868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"143'706","reported_seq":2977,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353002+0000","last_change":"2024-09-04T21:56:16.584413+0000","last_active":"2024-09-04T21:56:35.353002+0000","last_peered":"2024-09-04T21:56:35.353002+0000","last_clean":"2024-09-04T21:56:35.353002+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:35.353002+0000","last_undegraded":"2024-09-04T21:56:35.353002+0000","last_fullsized":"2024-09-04T21:56:35.353002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:01:21.655483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"143'383","reported_seq":2457,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653129+0000","last_change":"2024-09-04T21:56:16.488947+0000","last_active":"2024-09-04T21:56:34.653129+0000","last_peered":"2024-09-04T21:56:34.653129+0000","last_clean":"2024-09-04T21:56:34.653129+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:34.653129+0000","last_undegraded":"2024-09-04T21:56:34.653129+0000","last_fullsized":"2024-09-04T21:56:34.653129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:49.649633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"144'357","reported_seq":3597,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182710+0000","last_change":"2024-09-04T21:56:29.613277+0000","last_active":"2024-09-04T21:56:35.182710+0000","last_peered":"2024-09-04T21:56:35.182710+0000","last_clean":"2024-09-04T21:56:35.182710+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:35.182710+0000","last_undegraded":"2024-09-04T21:56:35.182710+0000","last_fullsized":"2024-09-04T21:56:35.182710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:00.762632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'419","reported_seq":11217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227512+0000","last_change":"2024-09-04T21:56:16.600636+0000","last_active":"2024-09-04T21:56:35.227512+0000","last_peered":"2024-09-04T21:56:35.227512+0000","last_clean":"2024-09-04T21:56:35.227512+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:35.227512+0000","last_undegraded":"2024-09-04T21:56:35.227512+0000","last_fullsized":"2024-09-04T21:56:35.227512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:53:20.412017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"143'394","reported_seq":1388,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227770+0000","last_change":"2024-09-04T21:56:16.601950+0000","last_active":"2024-09-04T21:56:35.227770+0000","last_peered":"2024-09-04T21:56:35.227770+0000","last_clean":"2024-09-04T21:56:35.227770+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:35.227770+0000","last_undegraded":"2024-09-04T21:56:35.227770+0000","last_fullsized":"2024-09-04T21:56:35.227770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:42.603261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"143'488","reported_seq":11950,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227095+0000","last_change":"2024-09-04T21:56:16.602325+0000","last_active":"2024-09-04T21:56:35.227095+0000","last_peered":"2024-09-04T21:56:35.227095+0000","last_clean":"2024-09-04T21:56:35.227095+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:35.227095+0000","last_undegraded":"2024-09-04T21:56:35.227095+0000","last_fullsized":"2024-09-04T21:56:35.227095+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:41:05.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"143'830","reported_seq":3709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352971+0000","last_change":"2024-09-04T21:56:16.584288+0000","last_active":"2024-09-04T21:56:35.352971+0000","last_peered":"2024-09-04T21:56:35.352971+0000","last_clean":"2024-09-04T21:56:35.352971+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:35.352971+0000","last_undegraded":"2024-09-04T21:56:35.352971+0000","last_fullsized":"2024-09-04T21:56:35.352971+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:03:59.014910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"144'579","reported_seq":6064,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227102+0000","last_change":"2024-09-04T21:56:16.598876+0000","last_active":"2024-09-04T21:56:35.227102+0000","last_peered":"2024-09-04T21:56:35.227102+0000","last_clean":"2024-09-04T21:56:35.227102+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:35.227102+0000","last_undegraded":"2024-09-04T21:56:35.227102+0000","last_fullsized":"2024-09-04T21:56:35.227102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:34:16.388010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"143'763","reported_seq":13635,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352207+0000","last_change":"2024-09-04T21:56:16.528688+0000","last_active":"2024-09-04T21:56:35.352207+0000","last_peered":"2024-09-04T21:56:35.352207+0000","last_clean":"2024-09-04T21:56:35.352207+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:35.352207+0000","last_undegraded":"2024-09-04T21:56:35.352207+0000","last_fullsized":"2024-09-04T21:56:35.352207+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:08.477339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"142'973","reported_seq":2644,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227285+0000","last_change":"2024-09-04T21:56:16.596597+0000","last_active":"2024-09-04T21:56:35.227285+0000","last_peered":"2024-09-04T21:56:35.227285+0000","last_clean":"2024-09-04T21:56:35.227285+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:35.227285+0000","last_undegraded":"2024-09-04T21:56:35.227285+0000","last_fullsized":"2024-09-04T21:56:35.227285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:19:55.348359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17483,"ondisk_log_size":17483,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566094,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39276,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332564,"statfs":{"total":96636764160,"available":96596545536,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1945600,"data_stored":1504066,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45548,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326292,"statfs":{"total":96636764160,"available":96590123008,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34072,"kb_used_data":1916,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337768,"statfs":{"total":96636764160,"available":96601874432,"internally_reserved":0,"allocated":1961984,"data_stored":1504066,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:39.358 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-04T21:56:39.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:39.642 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:39.659 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-04T21:56:38.472889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355036,"kb_used_data":15052,"kb_used_omap":46,"kb_used_meta":339729,"kb_avail":754619684,"statfs":{"total":773094113280,"available":772730556416,"internally_reserved":0,"allocated":15413248,"data_stored":8519606,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":347882642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001923"},"pg_stats":[{"pgid":"3.1f","version":"144'764","reported_seq":2993,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352518+0000","last_change":"2024-09-04T21:56:16.529040+0000","last_active":"2024-09-04T21:56:35.352518+0000","last_peered":"2024-09-04T21:56:35.352518+0000","last_clean":"2024-09-04T21:56:35.352518+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:35.352518+0000","last_undegraded":"2024-09-04T21:56:35.352518+0000","last_fullsized":"2024-09-04T21:56:35.352518+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:39:44.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"143'382","reported_seq":3537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653287+0000","last_change":"2024-09-04T21:56:16.490790+0000","last_active":"2024-09-04T21:56:34.653287+0000","last_peered":"2024-09-04T21:56:34.653287+0000","last_clean":"2024-09-04T21:56:34.653287+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:34.653287+0000","last_undegraded":"2024-09-04T21:56:34.653287+0000","last_fullsized":"2024-09-04T21:56:34.653287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:56:20.741571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"143'566","reported_seq":19161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861349+0000","last_change":"2024-09-04T21:56:16.556682+0000","last_active":"2024-09-04T21:56:34.861349+0000","last_peered":"2024-09-04T21:56:34.861349+0000","last_clean":"2024-09-04T21:56:34.861349+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:34.861349+0000","last_undegraded":"2024-09-04T21:56:34.861349+0000","last_fullsized":"2024-09-04T21:56:34.861349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:18:12.735862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"144'394","reported_seq":2484,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182638+0000","last_change":"2024-09-04T21:56:29.616344+0000","last_active":"2024-09-04T21:56:35.182638+0000","last_peered":"2024-09-04T21:56:35.182638+0000","last_clean":"2024-09-04T21:56:35.182638+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:35.182638+0000","last_undegraded":"2024-09-04T21:56:35.182638+0000","last_fullsized":"2024-09-04T21:56:35.182638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:43:32.577364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"143'290","reported_seq":1711,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352394+0000","last_change":"2024-09-04T21:56:16.527857+0000","last_active":"2024-09-04T21:56:35.352394+0000","last_peered":"2024-09-04T21:56:35.352394+0000","last_clean":"2024-09-04T21:56:35.352394+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:35.352394+0000","last_undegraded":"2024-09-04T21:56:35.352394+0000","last_fullsized":"2024-09-04T21:56:35.352394+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:15:56.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"143'957","reported_seq":17593,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652386+0000","last_change":"2024-09-04T21:56:16.488214+0000","last_active":"2024-09-04T21:56:34.652386+0000","last_peered":"2024-09-04T21:56:34.652386+0000","last_clean":"2024-09-04T21:56:34.652386+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:34.652386+0000","last_undegraded":"2024-09-04T21:56:34.652386+0000","last_fullsized":"2024-09-04T21:56:34.652386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:19.822098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"143'1213","reported_seq":3290,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654113+0000","last_change":"2024-09-04T21:56:16.491514+0000","last_active":"2024-09-04T21:56:34.654113+0000","last_peered":"2024-09-04T21:56:34.654113+0000","last_clean":"2024-09-04T21:56:34.654113+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:34.654113+0000","last_undegraded":"2024-09-04T21:56:34.654113+0000","last_fullsized":"2024-09-04T21:56:34.654113+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:21.020845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'402","reported_seq":4769,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653256+0000","last_change":"2024-09-04T21:56:16.488004+0000","last_active":"2024-09-04T21:56:34.653256+0000","last_peered":"2024-09-04T21:56:34.653256+0000","last_clean":"2024-09-04T21:56:34.653256+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:34.653256+0000","last_undegraded":"2024-09-04T21:56:34.653256+0000","last_fullsized":"2024-09-04T21:56:34.653256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:00.780840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"144'956","reported_seq":3245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352481+0000","last_change":"2024-09-04T21:56:16.528237+0000","last_active":"2024-09-04T21:56:35.352481+0000","last_peered":"2024-09-04T21:56:35.352481+0000","last_clean":"2024-09-04T21:56:35.352481+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:35.352481+0000","last_undegraded":"2024-09-04T21:56:35.352481+0000","last_fullsized":"2024-09-04T21:56:35.352481+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:49:16.916644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'357","reported_seq":2708,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.183014+0000","last_change":"2024-09-04T21:56:29.617384+0000","last_active":"2024-09-04T21:56:35.183014+0000","last_peered":"2024-09-04T21:56:35.183014+0000","last_clean":"2024-09-04T21:56:35.183014+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:35.183014+0000","last_undegraded":"2024-09-04T21:56:35.183014+0000","last_fullsized":"2024-09-04T21:56:35.183014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:30:08.502145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"144'393","reported_seq":3169,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227566+0000","last_change":"2024-09-04T21:56:16.600842+0000","last_active":"2024-09-04T21:56:35.227566+0000","last_peered":"2024-09-04T21:56:35.227566+0000","last_clean":"2024-09-04T21:56:35.227566+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:35.227566+0000","last_undegraded":"2024-09-04T21:56:35.227566+0000","last_fullsized":"2024-09-04T21:56:35.227566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:00.160039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"143'345","reported_seq":2075,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652357+0000","last_change":"2024-09-04T21:56:16.488308+0000","last_active":"2024-09-04T21:56:34.652357+0000","last_peered":"2024-09-04T21:56:34.652357+0000","last_clean":"2024-09-04T21:56:34.652357+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:34.652357+0000","last_undegraded":"2024-09-04T21:56:34.652357+0000","last_fullsized":"2024-09-04T21:56:34.652357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:24:16.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353101+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:35.353101+0000","last_peered":"2024-09-04T21:56:35.353101+0000","last_clean":"2024-09-04T21:56:35.353101+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:35.353101+0000","last_undegraded":"2024-09-04T21:56:35.353101+0000","last_fullsized":"2024-09-04T21:56:35.353101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"144'465","reported_seq":2549,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352500+0000","last_change":"2024-09-04T21:56:16.527617+0000","last_active":"2024-09-04T21:56:35.352500+0000","last_peered":"2024-09-04T21:56:35.352500+0000","last_clean":"2024-09-04T21:56:35.352500+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:35.352500+0000","last_undegraded":"2024-09-04T21:56:35.352500+0000","last_fullsized":"2024-09-04T21:56:35.352500+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:34:20.478729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654438+0000","last_change":"2024-09-04T21:55:50.373208+0000","last_active":"2024-09-04T21:56:34.654438+0000","last_peered":"2024-09-04T21:56:34.654438+0000","last_clean":"2024-09-04T21:56:34.654438+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:34.654438+0000","last_undegraded":"2024-09-04T21:56:34.654438+0000","last_fullsized":"2024-09-04T21:56:34.654438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:01:34.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"143'353","reported_seq":3575,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653090+0000","last_change":"2024-09-04T21:56:16.491647+0000","last_active":"2024-09-04T21:56:34.653090+0000","last_peered":"2024-09-04T21:56:34.653090+0000","last_clean":"2024-09-04T21:56:34.653090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:34.653090+0000","last_undegraded":"2024-09-04T21:56:34.653090+0000","last_fullsized":"2024-09-04T21:56:34.653090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:45:54.303973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227837+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:35.227837+0000","last_peered":"2024-09-04T21:56:35.227837+0000","last_clean":"2024-09-04T21:56:35.227837+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:35.227837+0000","last_undegraded":"2024-09-04T21:56:35.227837+0000","last_fullsized":"2024-09-04T21:56:35.227837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"143'330","reported_seq":1971,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654382+0000","last_change":"2024-09-04T21:56:16.492759+0000","last_active":"2024-09-04T21:56:34.654382+0000","last_peered":"2024-09-04T21:56:34.654382+0000","last_clean":"2024-09-04T21:56:34.654382+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:34.654382+0000","last_undegraded":"2024-09-04T21:56:34.654382+0000","last_fullsized":"2024-09-04T21:56:34.654382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:13:53.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654037+0000","last_change":"2024-09-04T21:56:01.380611+0000","last_active":"2024-09-04T21:56:34.654037+0000","last_peered":"2024-09-04T21:56:34.654037+0000","last_clean":"2024-09-04T21:56:34.654037+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:34.654037+0000","last_undegraded":"2024-09-04T21:56:34.654037+0000","last_fullsized":"2024-09-04T21:56:34.654037+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:44.108771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'560","reported_seq":11918,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182904+0000","last_change":"2024-09-04T21:56:29.614226+0000","last_active":"2024-09-04T21:56:35.182904+0000","last_peered":"2024-09-04T21:56:35.182904+0000","last_clean":"2024-09-04T21:56:35.182904+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:35.182904+0000","last_undegraded":"2024-09-04T21:56:35.182904+0000","last_fullsized":"2024-09-04T21:56:35.182904+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:09:39.718023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"144'509","reported_seq":4121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652418+0000","last_change":"2024-09-04T21:56:16.490969+0000","last_active":"2024-09-04T21:56:34.652418+0000","last_peered":"2024-09-04T21:56:34.652418+0000","last_clean":"2024-09-04T21:56:34.652418+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:34.652418+0000","last_undegraded":"2024-09-04T21:56:34.652418+0000","last_fullsized":"2024-09-04T21:56:34.652418+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:46:12.808214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182664+0000","last_change":"2024-09-04T21:56:29.610468+0000","last_active":"2024-09-04T21:56:35.182664+0000","last_peered":"2024-09-04T21:56:35.182664+0000","last_clean":"2024-09-04T21:56:35.182664+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:35.182664+0000","last_undegraded":"2024-09-04T21:56:35.182664+0000","last_fullsized":"2024-09-04T21:56:35.182664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:27:48.891402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"143'376","reported_seq":3305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653993+0000","last_change":"2024-09-04T21:56:16.493672+0000","last_active":"2024-09-04T21:56:34.653993+0000","last_peered":"2024-09-04T21:56:34.653993+0000","last_clean":"2024-09-04T21:56:34.653993+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:34.653993+0000","last_undegraded":"2024-09-04T21:56:34.653993+0000","last_fullsized":"2024-09-04T21:56:34.653993+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:44:32.875548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861461+0000","last_change":"2024-09-04T21:53:47.808149+0000","last_active":"2024-09-04T21:56:34.861461+0000","last_peered":"2024-09-04T21:56:34.861461+0000","last_clean":"2024-09-04T21:56:34.861461+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:34.861461+0000","last_undegraded":"2024-09-04T21:56:34.861461+0000","last_fullsized":"2024-09-04T21:56:34.861461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:34:52.946482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'618","reported_seq":4691,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352208+0000","last_change":"2024-09-04T21:56:16.527386+0000","last_active":"2024-09-04T21:56:35.352208+0000","last_peered":"2024-09-04T21:56:35.352208+0000","last_clean":"2024-09-04T21:56:35.352208+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:35.352208+0000","last_undegraded":"2024-09-04T21:56:35.352208+0000","last_fullsized":"2024-09-04T21:56:35.352208+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:25:55.616051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227578+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:35.227578+0000","last_peered":"2024-09-04T21:56:35.227578+0000","last_clean":"2024-09-04T21:56:35.227578+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:35.227578+0000","last_undegraded":"2024-09-04T21:56:35.227578+0000","last_fullsized":"2024-09-04T21:56:35.227578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"144'426","reported_seq":3648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652929+0000","last_change":"2024-09-04T21:56:16.487890+0000","last_active":"2024-09-04T21:56:34.652929+0000","last_peered":"2024-09-04T21:56:34.652929+0000","last_clean":"2024-09-04T21:56:34.652929+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:34.652929+0000","last_undegraded":"2024-09-04T21:56:34.652929+0000","last_fullsized":"2024-09-04T21:56:34.652929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:00:48.401698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":115,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182685+0000","last_change":"2024-09-04T21:56:29.613744+0000","last_active":"2024-09-04T21:56:35.182685+0000","last_peered":"2024-09-04T21:56:35.182685+0000","last_clean":"2024-09-04T21:56:35.182685+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:35.182685+0000","last_undegraded":"2024-09-04T21:56:35.182685+0000","last_fullsized":"2024-09-04T21:56:35.182685+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:26.686223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227261+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:35.227261+0000","last_peered":"2024-09-04T21:56:35.227261+0000","last_clean":"2024-09-04T21:56:35.227261+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:35.227261+0000","last_undegraded":"2024-09-04T21:56:35.227261+0000","last_fullsized":"2024-09-04T21:56:35.227261+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"144'590","reported_seq":3820,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652932+0000","last_change":"2024-09-04T21:56:16.489907+0000","last_active":"2024-09-04T21:56:34.652932+0000","last_peered":"2024-09-04T21:56:34.652932+0000","last_clean":"2024-09-04T21:56:34.652932+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:34.652932+0000","last_undegraded":"2024-09-04T21:56:34.652932+0000","last_fullsized":"2024-09-04T21:56:34.652932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:10:28.643680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"143'345","reported_seq":2203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652276+0000","last_change":"2024-09-04T21:56:16.491283+0000","last_active":"2024-09-04T21:56:34.652276+0000","last_peered":"2024-09-04T21:56:34.652276+0000","last_clean":"2024-09-04T21:56:34.652276+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:34.652276+0000","last_undegraded":"2024-09-04T21:56:34.652276+0000","last_fullsized":"2024-09-04T21:56:34.652276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:43:39.964868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"143'706","reported_seq":2977,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353002+0000","last_change":"2024-09-04T21:56:16.584413+0000","last_active":"2024-09-04T21:56:35.353002+0000","last_peered":"2024-09-04T21:56:35.353002+0000","last_clean":"2024-09-04T21:56:35.353002+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:35.353002+0000","last_undegraded":"2024-09-04T21:56:35.353002+0000","last_fullsized":"2024-09-04T21:56:35.353002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:01:21.655483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"143'383","reported_seq":2457,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653129+0000","last_change":"2024-09-04T21:56:16.488947+0000","last_active":"2024-09-04T21:56:34.653129+0000","last_peered":"2024-09-04T21:56:34.653129+0000","last_clean":"2024-09-04T21:56:34.653129+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:34.653129+0000","last_undegraded":"2024-09-04T21:56:34.653129+0000","last_fullsized":"2024-09-04T21:56:34.653129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:49.649633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"144'357","reported_seq":3597,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182710+0000","last_change":"2024-09-04T21:56:29.613277+0000","last_active":"2024-09-04T21:56:35.182710+0000","last_peered":"2024-09-04T21:56:35.182710+0000","last_clean":"2024-09-04T21:56:35.182710+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:35.182710+0000","last_undegraded":"2024-09-04T21:56:35.182710+0000","last_fullsized":"2024-09-04T21:56:35.182710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:00.762632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'419","reported_seq":11217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227512+0000","last_change":"2024-09-04T21:56:16.600636+0000","last_active":"2024-09-04T21:56:35.227512+0000","last_peered":"2024-09-04T21:56:35.227512+0000","last_clean":"2024-09-04T21:56:35.227512+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:35.227512+0000","last_undegraded":"2024-09-04T21:56:35.227512+0000","last_fullsized":"2024-09-04T21:56:35.227512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:53:20.412017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"143'394","reported_seq":1388,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227770+0000","last_change":"2024-09-04T21:56:16.601950+0000","last_active":"2024-09-04T21:56:35.227770+0000","last_peered":"2024-09-04T21:56:35.227770+0000","last_clean":"2024-09-04T21:56:35.227770+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:35.227770+0000","last_undegraded":"2024-09-04T21:56:35.227770+0000","last_fullsized":"2024-09-04T21:56:35.227770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:42.603261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"143'488","reported_seq":11950,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227095+0000","last_change":"2024-09-04T21:56:16.602325+0000","last_active":"2024-09-04T21:56:35.227095+0000","last_peered":"2024-09-04T21:56:35.227095+0000","last_clean":"2024-09-04T21:56:35.227095+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:35.227095+0000","last_undegraded":"2024-09-04T21:56:35.227095+0000","last_fullsized":"2024-09-04T21:56:35.227095+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:41:05.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"143'830","reported_seq":3709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352971+0000","last_change":"2024-09-04T21:56:16.584288+0000","last_active":"2024-09-04T21:56:35.352971+0000","last_peered":"2024-09-04T21:56:35.352971+0000","last_clean":"2024-09-04T21:56:35.352971+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:35.352971+0000","last_undegraded":"2024-09-04T21:56:35.352971+0000","last_fullsized":"2024-09-04T21:56:35.352971+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:03:59.014910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"144'579","reported_seq":6064,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227102+0000","last_change":"2024-09-04T21:56:16.598876+0000","last_active":"2024-09-04T21:56:35.227102+0000","last_peered":"2024-09-04T21:56:35.227102+0000","last_clean":"2024-09-04T21:56:35.227102+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:35.227102+0000","last_undegraded":"2024-09-04T21:56:35.227102+0000","last_fullsized":"2024-09-04T21:56:35.227102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:34:16.388010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"143'763","reported_seq":13635,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352207+0000","last_change":"2024-09-04T21:56:16.528688+0000","last_active":"2024-09-04T21:56:35.352207+0000","last_peered":"2024-09-04T21:56:35.352207+0000","last_clean":"2024-09-04T21:56:35.352207+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:35.352207+0000","last_undegraded":"2024-09-04T21:56:35.352207+0000","last_fullsized":"2024-09-04T21:56:35.352207+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:23:08.477339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"142'973","reported_seq":2644,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227285+0000","last_change":"2024-09-04T21:56:16.596597+0000","last_active":"2024-09-04T21:56:35.227285+0000","last_peered":"2024-09-04T21:56:35.227285+0000","last_clean":"2024-09-04T21:56:35.227285+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:35.227285+0000","last_undegraded":"2024-09-04T21:56:35.227285+0000","last_fullsized":"2024-09-04T21:56:35.227285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:19:55.348359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17483,"ondisk_log_size":17483,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1904640,"data_stored":913698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566094,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39276,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332564,"statfs":{"total":96636764160,"available":96596545536,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1945600,"data_stored":1504066,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45548,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326292,"statfs":{"total":96636764160,"available":96590123008,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34072,"kb_used_data":1916,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337768,"statfs":{"total":96636764160,"available":96601874432,"internally_reserved":0,"allocated":1961984,"data_stored":1504066,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:39.661 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-04T21:56:39.946 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:39.947 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":160,"fsid":"7c7b20bb-5983-49a4-8b2c-296a55ea8d95","created":"2024-09-04T21:53:25.631463+0000","modified":"2024-09-04T21:56:34.646237+0000","last_up_change":"2024-09-04T21:55:37.706985+0000","last_in_change":"2024-09-04T21:56:00.354804+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-04T21:53:36.459454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-04T21:53:43.849962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-04T21:53:54.481634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":36,"snap_epoch":144,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d210c577-fc3d-48c0-9acb-67afdd5a485a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6817","nonce":3808224845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6819","nonce":3808224845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6823","nonce":3808224845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3808224845},{"type":"v1","addr":"172.21.15.112:6821","nonce":3808224845}]},"public_addr":"172.21.15.112:6817/3808224845","cluster_addr":"172.21.15.112:6819/3808224845","heartbeat_back_addr":"172.21.15.112:6823/3808224845","heartbeat_front_addr":"172.21.15.112:6821/3808224845","state":["exists","up"]},{"osd":1,"uuid":"3d1c8d55-e5ac-4787-a725-6895bb2bf3f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6825","nonce":245081828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6827","nonce":245081828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6831","nonce":245081828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":245081828},{"type":"v1","addr":"172.21.15.112:6829","nonce":245081828}]},"public_addr":"172.21.15.112:6825/245081828","cluster_addr":"172.21.15.112:6827/245081828","heartbeat_back_addr":"172.21.15.112:6831/245081828","heartbeat_front_addr":"172.21.15.112:6829/245081828","state":["exists","up"]},{"osd":2,"uuid":"5a67b7d1-ef0a-4935-9af0-01389e12d43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6801","nonce":1590148387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6803","nonce":1590148387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6807","nonce":1590148387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1590148387},{"type":"v1","addr":"172.21.15.112:6805","nonce":1590148387}]},"public_addr":"172.21.15.112:6801/1590148387","cluster_addr":"172.21.15.112:6803/1590148387","heartbeat_back_addr":"172.21.15.112:6807/1590148387","heartbeat_front_addr":"172.21.15.112:6805/1590148387","state":["exists","up"]},{"osd":3,"uuid":"0b3ec943-bdb7-47e3-bcca-b52070713e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6809","nonce":2461796488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6811","nonce":2461796488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6815","nonce":2461796488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2461796488},{"type":"v1","addr":"172.21.15.112:6813","nonce":2461796488}]},"public_addr":"172.21.15.112:6809/2461796488","cluster_addr":"172.21.15.112:6811/2461796488","heartbeat_back_addr":"172.21.15.112:6815/2461796488","heartbeat_front_addr":"172.21.15.112:6813/2461796488","state":["exists","up"]},{"osd":4,"uuid":"09964dd2-405d-483d-822d-1b8c74497413","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":84,"up_from":105,"up_thru":137,"down_at":85,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6825","nonce":3575407730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6827","nonce":3575407730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6831","nonce":3575407730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3575407730},{"type":"v1","addr":"172.21.15.157:6829","nonce":3575407730}]},"public_addr":"172.21.15.157:6825/3575407730","cluster_addr":"172.21.15.157:6827/3575407730","heartbeat_back_addr":"172.21.15.157:6831/3575407730","heartbeat_front_addr":"172.21.15.157:6829/3575407730","state":["exists","up"]},{"osd":5,"uuid":"87306896-7b79-487f-ab06-ff337bb72f37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6809","nonce":352715412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6811","nonce":352715412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6815","nonce":352715412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":352715412},{"type":"v1","addr":"172.21.15.157:6813","nonce":352715412}]},"public_addr":"172.21.15.157:6809/352715412","cluster_addr":"172.21.15.157:6811/352715412","heartbeat_back_addr":"172.21.15.157:6815/352715412","heartbeat_front_addr":"172.21.15.157:6813/352715412","state":["exists","up"]},{"osd":6,"uuid":"e7948c3b-e9b4-4a9b-99be-8364f09b8c1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6801","nonce":3465983375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6803","nonce":3465983375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6808","nonce":3465983375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3465983375},{"type":"v1","addr":"172.21.15.157:6805","nonce":3465983375}]},"public_addr":"172.21.15.157:6801/3465983375","cluster_addr":"172.21.15.157:6803/3465983375","heartbeat_back_addr":"172.21.15.157:6808/3465983375","heartbeat_front_addr":"172.21.15.157:6805/3465983375","state":["exists","up"]},{"osd":7,"uuid":"c9fdaad2-f75b-4dcd-a170-3e346fda0e16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":21,"up_from":48,"up_thru":128,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6817","nonce":471611328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6819","nonce":471611328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6823","nonce":471611328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":471611328},{"type":"v1","addr":"172.21.15.157:6821","nonce":471611328}]},"public_addr":"172.21.15.157:6817/471611328","cluster_addr":"172.21.15.157:6819/471611328","heartbeat_back_addr":"172.21.15.157:6823/471611328","heartbeat_front_addr":"172.21.15.157:6821/471611328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.725388+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:35.420675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.389632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.611983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.796105+0000","dead_epoch":85},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.383432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.731214+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-04T21:53:34.469040+0000","dead_epoch":22}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-04T21:56:40.965 INFO:tasks.ceph:Scrubbing osd.0 2024-09-04T21:56:40.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:41.128 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:41.129 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:41.129 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:41.144 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-04T21:56:41.434 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 0 to deep-scrub 2024-09-04T21:56:41.452 INFO:tasks.ceph:Scrubbing osd.1 2024-09-04T21:56:41.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:41.591 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:41.591 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:41.591 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:41.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-04T21:56:41.895 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 1 to deep-scrub 2024-09-04T21:56:41.914 INFO:tasks.ceph:Scrubbing osd.2 2024-09-04T21:56:41.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:42.062 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:42.062 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:42.062 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:42.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-04T21:56:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 2 to deep-scrub 2024-09-04T21:56:42.376 INFO:tasks.ceph:Scrubbing osd.3 2024-09-04T21:56:42.376 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:42.523 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:42.523 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:42.523 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:42.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-04T21:56:42.830 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 3 to deep-scrub 2024-09-04T21:56:42.849 INFO:tasks.ceph:Scrubbing osd.4 2024-09-04T21:56:42.849 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:42.998 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:42.999 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:42.999 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:43.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-04T21:56:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 4 to deep-scrub 2024-09-04T21:56:43.336 INFO:tasks.ceph:Scrubbing osd.5 2024-09-04T21:56:43.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:43.484 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:43.484 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:43.484 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:43.499 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-04T21:56:43.791 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 5 to deep-scrub 2024-09-04T21:56:43.811 INFO:tasks.ceph:Scrubbing osd.6 2024-09-04T21:56:43.811 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:43.963 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:43.963 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:43.963 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:43.979 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-04T21:56:44.273 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 6 to deep-scrub 2024-09-04T21:56:44.291 INFO:tasks.ceph:Scrubbing osd.7 2024-09-04T21:56:44.291 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-04T21:56:44.443 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-04T21:56:44.443 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-04T21:56:44.443 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-04T21:56:44.458 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-04T21:56:44.758 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 7 to deep-scrub 2024-09-04T21:56:44.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:56:45.075 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:56:45.075 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:56:45.091 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-04T21:56:44.474510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17570,"ondisk_log_size":17570,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355032,"kb_used_data":15088,"kb_used_omap":46,"kb_used_meta":339729,"kb_avail":754619688,"statfs":{"total":773094113280,"available":772730560512,"internally_reserved":0,"allocated":15450112,"data_stored":8537132,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":347882642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002967"},"pg_stats":[{"pgid":"3.1f","version":"144'764","reported_seq":2993,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352518+0000","last_change":"2024-09-04T21:56:16.529040+0000","last_active":"2024-09-04T21:56:35.352518+0000","last_peered":"2024-09-04T21:56:35.352518+0000","last_clean":"2024-09-04T21:56:35.352518+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:35.352518+0000","last_undegraded":"2024-09-04T21:56:35.352518+0000","last_fullsized":"2024-09-04T21:56:35.352518+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:39:44.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"143'382","reported_seq":3537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653287+0000","last_change":"2024-09-04T21:56:16.490790+0000","last_active":"2024-09-04T21:56:34.653287+0000","last_peered":"2024-09-04T21:56:34.653287+0000","last_clean":"2024-09-04T21:56:34.653287+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:34.653287+0000","last_undegraded":"2024-09-04T21:56:34.653287+0000","last_fullsized":"2024-09-04T21:56:34.653287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:56:20.741571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"143'566","reported_seq":19161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.861349+0000","last_change":"2024-09-04T21:56:16.556682+0000","last_active":"2024-09-04T21:56:34.861349+0000","last_peered":"2024-09-04T21:56:34.861349+0000","last_clean":"2024-09-04T21:56:34.861349+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:34.861349+0000","last_undegraded":"2024-09-04T21:56:34.861349+0000","last_fullsized":"2024-09-04T21:56:34.861349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:18:12.735862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"144'394","reported_seq":2484,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182638+0000","last_change":"2024-09-04T21:56:29.616344+0000","last_active":"2024-09-04T21:56:35.182638+0000","last_peered":"2024-09-04T21:56:35.182638+0000","last_clean":"2024-09-04T21:56:35.182638+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:35.182638+0000","last_undegraded":"2024-09-04T21:56:35.182638+0000","last_fullsized":"2024-09-04T21:56:35.182638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:43:32.577364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"143'290","reported_seq":1711,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352394+0000","last_change":"2024-09-04T21:56:16.527857+0000","last_active":"2024-09-04T21:56:35.352394+0000","last_peered":"2024-09-04T21:56:35.352394+0000","last_clean":"2024-09-04T21:56:35.352394+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:35.352394+0000","last_undegraded":"2024-09-04T21:56:35.352394+0000","last_fullsized":"2024-09-04T21:56:35.352394+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:15:56.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"143'957","reported_seq":17593,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652386+0000","last_change":"2024-09-04T21:56:16.488214+0000","last_active":"2024-09-04T21:56:34.652386+0000","last_peered":"2024-09-04T21:56:34.652386+0000","last_clean":"2024-09-04T21:56:34.652386+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:34.652386+0000","last_undegraded":"2024-09-04T21:56:34.652386+0000","last_fullsized":"2024-09-04T21:56:34.652386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:31:19.822098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"143'1213","reported_seq":3290,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654113+0000","last_change":"2024-09-04T21:56:16.491514+0000","last_active":"2024-09-04T21:56:34.654113+0000","last_peered":"2024-09-04T21:56:34.654113+0000","last_clean":"2024-09-04T21:56:34.654113+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:34.654113+0000","last_undegraded":"2024-09-04T21:56:34.654113+0000","last_fullsized":"2024-09-04T21:56:34.654113+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:27:21.020845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'402","reported_seq":4769,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653256+0000","last_change":"2024-09-04T21:56:16.488004+0000","last_active":"2024-09-04T21:56:34.653256+0000","last_peered":"2024-09-04T21:56:34.653256+0000","last_clean":"2024-09-04T21:56:34.653256+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:34.653256+0000","last_undegraded":"2024-09-04T21:56:34.653256+0000","last_fullsized":"2024-09-04T21:56:34.653256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:10:00.780840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"144'956","reported_seq":3245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352481+0000","last_change":"2024-09-04T21:56:16.528237+0000","last_active":"2024-09-04T21:56:35.352481+0000","last_peered":"2024-09-04T21:56:35.352481+0000","last_clean":"2024-09-04T21:56:35.352481+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:35.352481+0000","last_undegraded":"2024-09-04T21:56:35.352481+0000","last_fullsized":"2024-09-04T21:56:35.352481+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:49:16.916644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"142'357","reported_seq":2708,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.183014+0000","last_change":"2024-09-04T21:56:29.617384+0000","last_active":"2024-09-04T21:56:35.183014+0000","last_peered":"2024-09-04T21:56:35.183014+0000","last_clean":"2024-09-04T21:56:35.183014+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:35.183014+0000","last_undegraded":"2024-09-04T21:56:35.183014+0000","last_fullsized":"2024-09-04T21:56:35.183014+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:30:08.502145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"144'393","reported_seq":3169,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227566+0000","last_change":"2024-09-04T21:56:16.600842+0000","last_active":"2024-09-04T21:56:35.227566+0000","last_peered":"2024-09-04T21:56:35.227566+0000","last_clean":"2024-09-04T21:56:35.227566+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:35.227566+0000","last_undegraded":"2024-09-04T21:56:35.227566+0000","last_fullsized":"2024-09-04T21:56:35.227566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:16:00.160039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"143'345","reported_seq":2075,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652357+0000","last_change":"2024-09-04T21:56:16.488308+0000","last_active":"2024-09-04T21:56:34.652357+0000","last_peered":"2024-09-04T21:56:34.652357+0000","last_clean":"2024-09-04T21:56:34.652357+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:34.652357+0000","last_undegraded":"2024-09-04T21:56:34.652357+0000","last_fullsized":"2024-09-04T21:56:34.652357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:24:16.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353101+0000","last_change":"2024-09-04T21:54:31.581633+0000","last_active":"2024-09-04T21:56:35.353101+0000","last_peered":"2024-09-04T21:56:35.353101+0000","last_clean":"2024-09-04T21:56:35.353101+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:35.353101+0000","last_undegraded":"2024-09-04T21:56:35.353101+0000","last_fullsized":"2024-09-04T21:56:35.353101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:17.051767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"144'465","reported_seq":2549,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352500+0000","last_change":"2024-09-04T21:56:16.527617+0000","last_active":"2024-09-04T21:56:35.352500+0000","last_peered":"2024-09-04T21:56:35.352500+0000","last_clean":"2024-09-04T21:56:35.352500+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:35.352500+0000","last_undegraded":"2024-09-04T21:56:35.352500+0000","last_fullsized":"2024-09-04T21:56:35.352500+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:34:20.478729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":102,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.474646+0000","last_change":"2024-09-04T21:56:43.474646+0000","last_active":"2024-09-04T21:56:43.474646+0000","last_peered":"2024-09-04T21:56:43.474646+0000","last_clean":"2024-09-04T21:56:43.474646+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:43.474646+0000","last_undegraded":"2024-09-04T21:56:43.474646+0000","last_fullsized":"2024-09-04T21:56:43.474646+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:43.474563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:43.474563+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.474563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:24:16.496371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"143'353","reported_seq":3575,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653090+0000","last_change":"2024-09-04T21:56:16.491647+0000","last_active":"2024-09-04T21:56:34.653090+0000","last_peered":"2024-09-04T21:56:34.653090+0000","last_clean":"2024-09-04T21:56:34.653090+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:34.653090+0000","last_undegraded":"2024-09-04T21:56:34.653090+0000","last_fullsized":"2024-09-04T21:56:34.653090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:45:54.303973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227837+0000","last_change":"2024-09-04T21:56:01.383765+0000","last_active":"2024-09-04T21:56:35.227837+0000","last_peered":"2024-09-04T21:56:35.227837+0000","last_clean":"2024-09-04T21:56:35.227837+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:35.227837+0000","last_undegraded":"2024-09-04T21:56:35.227837+0000","last_fullsized":"2024-09-04T21:56:35.227837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:00:58.650164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"143'330","reported_seq":1971,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.654382+0000","last_change":"2024-09-04T21:56:16.492759+0000","last_active":"2024-09-04T21:56:34.654382+0000","last_peered":"2024-09-04T21:56:34.654382+0000","last_clean":"2024-09-04T21:56:34.654382+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:34.654382+0000","last_undegraded":"2024-09-04T21:56:34.654382+0000","last_fullsized":"2024-09-04T21:56:34.654382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:13:53.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":108,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.506849+0000","last_change":"2024-09-04T21:56:42.506849+0000","last_active":"2024-09-04T21:56:42.506849+0000","last_peered":"2024-09-04T21:56:42.506849+0000","last_clean":"2024-09-04T21:56:42.506849+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:42.506849+0000","last_undegraded":"2024-09-04T21:56:42.506849+0000","last_fullsized":"2024-09-04T21:56:42.506849+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:42.506758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:42.506758+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.506758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:26:15.610554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'560","reported_seq":11918,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182904+0000","last_change":"2024-09-04T21:56:29.614226+0000","last_active":"2024-09-04T21:56:35.182904+0000","last_peered":"2024-09-04T21:56:35.182904+0000","last_clean":"2024-09-04T21:56:35.182904+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:35.182904+0000","last_undegraded":"2024-09-04T21:56:35.182904+0000","last_fullsized":"2024-09-04T21:56:35.182904+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:09:39.718023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"144'509","reported_seq":4121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652418+0000","last_change":"2024-09-04T21:56:16.490969+0000","last_active":"2024-09-04T21:56:34.652418+0000","last_peered":"2024-09-04T21:56:34.652418+0000","last_clean":"2024-09-04T21:56:34.652418+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:34.652418+0000","last_undegraded":"2024-09-04T21:56:34.652418+0000","last_fullsized":"2024-09-04T21:56:34.652418+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:46:12.808214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182664+0000","last_change":"2024-09-04T21:56:29.610468+0000","last_active":"2024-09-04T21:56:35.182664+0000","last_peered":"2024-09-04T21:56:35.182664+0000","last_clean":"2024-09-04T21:56:35.182664+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:35.182664+0000","last_undegraded":"2024-09-04T21:56:35.182664+0000","last_fullsized":"2024-09-04T21:56:35.182664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:27:48.891402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"143'376","reported_seq":3305,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653993+0000","last_change":"2024-09-04T21:56:16.493672+0000","last_active":"2024-09-04T21:56:34.653993+0000","last_peered":"2024-09-04T21:56:34.653993+0000","last_clean":"2024-09-04T21:56:34.653993+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:34.653993+0000","last_undegraded":"2024-09-04T21:56:34.653993+0000","last_fullsized":"2024-09-04T21:56:34.653993+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:44:32.875548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.714491+0000","last_change":"2024-09-04T21:56:42.714491+0000","last_active":"2024-09-04T21:56:42.714491+0000","last_peered":"2024-09-04T21:56:42.714491+0000","last_clean":"2024-09-04T21:56:42.714491+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:42.714491+0000","last_undegraded":"2024-09-04T21:56:42.714491+0000","last_fullsized":"2024-09-04T21:56:42.714491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:42.714384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:42.714384+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.714384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:34:18.287606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'625","reported_seq":4706,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.107231+0000","last_change":"2024-09-04T21:56:42.107231+0000","last_active":"2024-09-04T21:56:42.107231+0000","last_peered":"2024-09-04T21:56:42.107231+0000","last_clean":"2024-09-04T21:56:42.107231+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:42.107231+0000","last_undegraded":"2024-09-04T21:56:42.107231+0000","last_fullsized":"2024-09-04T21:56:42.107231+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'625","last_scrub_stamp":"2024-09-04T21:56:42.107144+0000","last_deep_scrub":"160'625","last_deep_scrub_stamp":"2024-09-04T21:56:42.107144+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.107144+0000","objects_scrubbed":7,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:22:32.286933+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227578+0000","last_change":"2024-09-04T21:55:50.373568+0000","last_active":"2024-09-04T21:56:35.227578+0000","last_peered":"2024-09-04T21:56:35.227578+0000","last_clean":"2024-09-04T21:56:35.227578+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:35.227578+0000","last_undegraded":"2024-09-04T21:56:35.227578+0000","last_fullsized":"2024-09-04T21:56:35.227578+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:48:07.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"144'426","reported_seq":3648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652929+0000","last_change":"2024-09-04T21:56:16.487890+0000","last_active":"2024-09-04T21:56:34.652929+0000","last_peered":"2024-09-04T21:56:34.652929+0000","last_clean":"2024-09-04T21:56:34.652929+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:34.652929+0000","last_undegraded":"2024-09-04T21:56:34.652929+0000","last_fullsized":"2024-09-04T21:56:34.652929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:00:48.401698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":115,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182685+0000","last_change":"2024-09-04T21:56:29.613744+0000","last_active":"2024-09-04T21:56:35.182685+0000","last_peered":"2024-09-04T21:56:35.182685+0000","last_clean":"2024-09-04T21:56:35.182685+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:35.182685+0000","last_undegraded":"2024-09-04T21:56:35.182685+0000","last_fullsized":"2024-09-04T21:56:35.182685+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:44.760731+0000","last_clean_scrub_stamp":"2024-09-04T21:53:44.760731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:15:26.686223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227261+0000","last_change":"2024-09-04T21:56:01.393548+0000","last_active":"2024-09-04T21:56:35.227261+0000","last_peered":"2024-09-04T21:56:35.227261+0000","last_clean":"2024-09-04T21:56:35.227261+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:35.227261+0000","last_undegraded":"2024-09-04T21:56:35.227261+0000","last_fullsized":"2024-09-04T21:56:35.227261+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:36.701304+0000","last_clean_scrub_stamp":"2024-09-04T21:53:36.701304+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:08:43.636908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"144'590","reported_seq":3820,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652932+0000","last_change":"2024-09-04T21:56:16.489907+0000","last_active":"2024-09-04T21:56:34.652932+0000","last_peered":"2024-09-04T21:56:34.652932+0000","last_clean":"2024-09-04T21:56:34.652932+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:34.652932+0000","last_undegraded":"2024-09-04T21:56:34.652932+0000","last_fullsized":"2024-09-04T21:56:34.652932+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:10:28.643680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"143'345","reported_seq":2203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.652276+0000","last_change":"2024-09-04T21:56:16.491283+0000","last_active":"2024-09-04T21:56:34.652276+0000","last_peered":"2024-09-04T21:56:34.652276+0000","last_clean":"2024-09-04T21:56:34.652276+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:34.652276+0000","last_undegraded":"2024-09-04T21:56:34.652276+0000","last_fullsized":"2024-09-04T21:56:34.652276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:43:39.964868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"143'706","reported_seq":2977,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.353002+0000","last_change":"2024-09-04T21:56:16.584413+0000","last_active":"2024-09-04T21:56:35.353002+0000","last_peered":"2024-09-04T21:56:35.353002+0000","last_clean":"2024-09-04T21:56:35.353002+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:35.353002+0000","last_undegraded":"2024-09-04T21:56:35.353002+0000","last_fullsized":"2024-09-04T21:56:35.353002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:01:21.655483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"143'383","reported_seq":2457,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:34.653129+0000","last_change":"2024-09-04T21:56:16.488947+0000","last_active":"2024-09-04T21:56:34.653129+0000","last_peered":"2024-09-04T21:56:34.653129+0000","last_clean":"2024-09-04T21:56:34.653129+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:34.653129+0000","last_undegraded":"2024-09-04T21:56:34.653129+0000","last_fullsized":"2024-09-04T21:56:34.653129+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:49.649633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"144'357","reported_seq":3597,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.182710+0000","last_change":"2024-09-04T21:56:29.613277+0000","last_active":"2024-09-04T21:56:35.182710+0000","last_peered":"2024-09-04T21:56:35.182710+0000","last_clean":"2024-09-04T21:56:35.182710+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:35.182710+0000","last_undegraded":"2024-09-04T21:56:35.182710+0000","last_fullsized":"2024-09-04T21:56:35.182710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:52:00.762632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'419","reported_seq":11217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227512+0000","last_change":"2024-09-04T21:56:16.600636+0000","last_active":"2024-09-04T21:56:35.227512+0000","last_peered":"2024-09-04T21:56:35.227512+0000","last_clean":"2024-09-04T21:56:35.227512+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:35.227512+0000","last_undegraded":"2024-09-04T21:56:35.227512+0000","last_fullsized":"2024-09-04T21:56:35.227512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:53:20.412017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"143'394","reported_seq":1388,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227770+0000","last_change":"2024-09-04T21:56:16.601950+0000","last_active":"2024-09-04T21:56:35.227770+0000","last_peered":"2024-09-04T21:56:35.227770+0000","last_clean":"2024-09-04T21:56:35.227770+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:35.227770+0000","last_undegraded":"2024-09-04T21:56:35.227770+0000","last_fullsized":"2024-09-04T21:56:35.227770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:14:42.603261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"143'488","reported_seq":11950,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227095+0000","last_change":"2024-09-04T21:56:16.602325+0000","last_active":"2024-09-04T21:56:35.227095+0000","last_peered":"2024-09-04T21:56:35.227095+0000","last_clean":"2024-09-04T21:56:35.227095+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:35.227095+0000","last_undegraded":"2024-09-04T21:56:35.227095+0000","last_fullsized":"2024-09-04T21:56:35.227095+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:41:05.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"143'830","reported_seq":3709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.352971+0000","last_change":"2024-09-04T21:56:16.584288+0000","last_active":"2024-09-04T21:56:35.352971+0000","last_peered":"2024-09-04T21:56:35.352971+0000","last_clean":"2024-09-04T21:56:35.352971+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:35.352971+0000","last_undegraded":"2024-09-04T21:56:35.352971+0000","last_fullsized":"2024-09-04T21:56:35.352971+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:03:59.014910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"144'579","reported_seq":6064,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227102+0000","last_change":"2024-09-04T21:56:16.598876+0000","last_active":"2024-09-04T21:56:35.227102+0000","last_peered":"2024-09-04T21:56:35.227102+0000","last_clean":"2024-09-04T21:56:35.227102+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:35.227102+0000","last_undegraded":"2024-09-04T21:56:35.227102+0000","last_fullsized":"2024-09-04T21:56:35.227102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:34:16.388010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"160'768","reported_seq":13648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.101463+0000","last_change":"2024-09-04T21:56:43.101463+0000","last_active":"2024-09-04T21:56:43.101463+0000","last_peered":"2024-09-04T21:56:43.101463+0000","last_clean":"2024-09-04T21:56:43.101463+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:43.101463+0000","last_undegraded":"2024-09-04T21:56:43.101463+0000","last_fullsized":"2024-09-04T21:56:43.101463+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'768","last_scrub_stamp":"2024-09-04T21:56:43.101381+0000","last_deep_scrub":"160'768","last_deep_scrub_stamp":"2024-09-04T21:56:43.101381+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.101381+0000","objects_scrubbed":5,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:18:07.772307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"142'973","reported_seq":2644,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:35.227285+0000","last_change":"2024-09-04T21:56:16.596597+0000","last_active":"2024-09-04T21:56:35.227285+0000","last_peered":"2024-09-04T21:56:35.227285+0000","last_clean":"2024-09-04T21:56:35.227285+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:35.227285+0000","last_undegraded":"2024-09-04T21:56:35.227285+0000","last_fullsized":"2024-09-04T21:56:35.227285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:53:55.032228+0000","last_clean_scrub_stamp":"2024-09-04T21:53:55.032228+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:19:55.348359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17495,"ondisk_log_size":17495,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28900,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342940,"statfs":{"total":96636764160,"available":96607170560,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566095,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39276,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332564,"statfs":{"total":96636764160,"available":96596545536,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509483,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1957888,"data_stored":1509908,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45548,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326292,"statfs":{"total":96636764160,"available":96590123008,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430236,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34076,"kb_used_data":1928,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337764,"statfs":{"total":96636764160,"available":96601870336,"internally_reserved":0,"allocated":1974272,"data_stored":1509908,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:56:45.097 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.097 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.097 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.097 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.097 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.097 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.098 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.098 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.098 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.098 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.098 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.098 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-04T21:53:44.760731+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=44, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-04T21:53:44.760731+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=44, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-04T21:53:44.760731+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=44, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.099 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-04T21:53:44.760731+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=44, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-04T21:53:44.760731+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=44, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-04T21:53:36.701304+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=36, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.100 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-04T21:53:55.032228+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=248, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=4, tm_hour=21, tm_min=56, tm_sec=39, tm_wday=2, tm_yday=248, tm_isdst=0) 2024-09-04T21:56:45.101 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-04T21:57:05.103 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-04T21:57:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-04T21:57:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-04T21:57:05.438 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-04T21:57:04.480550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":233,"num_object_clones":0,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":225,"num_read":192379,"num_read_kb":673948,"num_write":20825,"num_write_kb":316043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":90984613,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17789,"ondisk_log_size":17789,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":359152,"kb_used_data":15088,"kb_used_omap":46,"kb_used_meta":343889,"kb_avail":754615568,"statfs":{"total":773094113280,"available":772726341632,"internally_reserved":0,"allocated":15450112,"data_stored":8537132,"data_compressed":12659,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":47982,"internal_metadata":352142482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003722"},"pg_stats":[{"pgid":"3.1f","version":"160'773","reported_seq":3010,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:47.102251+0000","last_change":"2024-09-04T21:56:47.102251+0000","last_active":"2024-09-04T21:56:47.102251+0000","last_peered":"2024-09-04T21:56:47.102251+0000","last_clean":"2024-09-04T21:56:47.102251+0000","last_became_active":"2024-09-04T21:56:04.388881+0000","last_became_peered":"2024-09-04T21:56:04.388881+0000","last_unstale":"2024-09-04T21:56:47.102251+0000","last_undegraded":"2024-09-04T21:56:47.102251+0000","last_fullsized":"2024-09-04T21:56:47.102251+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'773","last_scrub_stamp":"2024-09-04T21:56:47.101988+0000","last_deep_scrub":"160'773","last_deep_scrub_stamp":"2024-09-04T21:56:47.101988+0000","last_clean_scrub_stamp":"2024-09-04T21:56:47.101988+0000","objects_scrubbed":9,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:06:02.474207+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.001790353,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5093,"num_read_kb":7784,"num_write":1087,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":614927,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"160'392","reported_seq":3555,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.823878+0000","last_change":"2024-09-04T21:56:46.823878+0000","last_active":"2024-09-04T21:56:46.823878+0000","last_peered":"2024-09-04T21:56:46.823878+0000","last_clean":"2024-09-04T21:56:46.823878+0000","last_became_active":"2024-09-04T21:53:55.349193+0000","last_became_peered":"2024-09-04T21:53:55.349193+0000","last_unstale":"2024-09-04T21:56:46.823878+0000","last_undegraded":"2024-09-04T21:56:46.823878+0000","last_fullsized":"2024-09-04T21:56:46.823878+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'392","last_scrub_stamp":"2024-09-04T21:56:46.823664+0000","last_deep_scrub":"160'392","last_deep_scrub_stamp":"2024-09-04T21:56:46.823664+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.823664+0000","objects_scrubbed":10,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:26:51.919450+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0051351749999999996,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3612,"num_read_kb":13268,"num_write":347,"num_write_kb":6351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"160'570","reported_seq":19173,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.740600+0000","last_change":"2024-09-04T21:56:43.740600+0000","last_active":"2024-09-04T21:56:43.740600+0000","last_peered":"2024-09-04T21:56:43.740600+0000","last_clean":"2024-09-04T21:56:43.740600+0000","last_became_active":"2024-09-04T21:56:11.373320+0000","last_became_peered":"2024-09-04T21:56:11.373320+0000","last_unstale":"2024-09-04T21:56:43.740600+0000","last_undegraded":"2024-09-04T21:56:43.740600+0000","last_fullsized":"2024-09-04T21:56:43.740600+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'570","last_scrub_stamp":"2024-09-04T21:56:43.740403+0000","last_deep_scrub":"160'570","last_deep_scrub_stamp":"2024-09-04T21:56:43.740403+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.740403+0000","objects_scrubbed":4,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:21:24.038889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00055601999999999995,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":19929,"num_read_kb":23239,"num_write":727,"num_write_kb":9172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1377719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"160'402","reported_seq":2500,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:49.026107+0000","last_change":"2024-09-04T21:56:49.026107+0000","last_active":"2024-09-04T21:56:49.026107+0000","last_peered":"2024-09-04T21:56:49.026107+0000","last_clean":"2024-09-04T21:56:49.026107+0000","last_became_active":"2024-09-04T21:56:29.613546+0000","last_became_peered":"2024-09-04T21:56:29.613546+0000","last_unstale":"2024-09-04T21:56:49.026107+0000","last_undegraded":"2024-09-04T21:56:49.026107+0000","last_fullsized":"2024-09-04T21:56:49.026107+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'402","last_scrub_stamp":"2024-09-04T21:56:49.025908+0000","last_deep_scrub":"160'402","last_deep_scrub_stamp":"2024-09-04T21:56:49.025908+0000","last_clean_scrub_stamp":"2024-09-04T21:56:49.025908+0000","objects_scrubbed":8,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:49:42.966481+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0075704960000000003,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2220,"num_read_kb":21141,"num_write":389,"num_write_kb":9222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":59643,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"160'296","reported_seq":1725,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.142007+0000","last_change":"2024-09-04T21:56:46.142007+0000","last_active":"2024-09-04T21:56:46.142007+0000","last_peered":"2024-09-04T21:56:46.142007+0000","last_clean":"2024-09-04T21:56:46.142007+0000","last_became_active":"2024-09-04T21:56:04.391640+0000","last_became_peered":"2024-09-04T21:56:04.391640+0000","last_unstale":"2024-09-04T21:56:46.142007+0000","last_undegraded":"2024-09-04T21:56:46.142007+0000","last_fullsized":"2024-09-04T21:56:46.142007+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'296","last_scrub_stamp":"2024-09-04T21:56:46.141781+0000","last_deep_scrub":"160'296","last_deep_scrub_stamp":"2024-09-04T21:56:46.141781+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.141781+0000","objects_scrubbed":6,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:29:09.751413+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00114478,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2273,"num_read_kb":17543,"num_write":269,"num_write_kb":10231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":4331336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"160'966","reported_seq":17610,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.591299+0000","last_change":"2024-09-04T21:56:46.591299+0000","last_active":"2024-09-04T21:56:46.591299+0000","last_peered":"2024-09-04T21:56:46.591299+0000","last_clean":"2024-09-04T21:56:46.591299+0000","last_became_active":"2024-09-04T21:55:53.466287+0000","last_became_peered":"2024-09-04T21:55:53.466287+0000","last_unstale":"2024-09-04T21:56:46.591299+0000","last_undegraded":"2024-09-04T21:56:46.591299+0000","last_fullsized":"2024-09-04T21:56:46.591299+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'966","last_scrub_stamp":"2024-09-04T21:56:46.591062+0000","last_deep_scrub":"160'966","last_deep_scrub_stamp":"2024-09-04T21:56:46.591062+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.591062+0000","objects_scrubbed":9,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:58:39.100428+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.002918478,"stat_sum":{"num_bytes":19,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":8,"num_read":19972,"num_read_kb":57190,"num_write":1222,"num_write_kb":17253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":6123307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"160'1219","reported_seq":3304,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:44.486361+0000","last_change":"2024-09-04T21:56:44.486361+0000","last_active":"2024-09-04T21:56:44.486361+0000","last_peered":"2024-09-04T21:56:44.486361+0000","last_clean":"2024-09-04T21:56:44.486361+0000","last_became_active":"2024-09-04T21:55:54.485552+0000","last_became_peered":"2024-09-04T21:55:54.485552+0000","last_unstale":"2024-09-04T21:56:44.486361+0000","last_undegraded":"2024-09-04T21:56:44.486361+0000","last_fullsized":"2024-09-04T21:56:44.486361+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'1219","last_scrub_stamp":"2024-09-04T21:56:44.486285+0000","last_deep_scrub":"160'1219","last_deep_scrub_stamp":"2024-09-04T21:56:44.486285+0000","last_clean_scrub_stamp":"2024-09-04T21:56:44.486285+0000","objects_scrubbed":6,"log_size":1219,"log_dups_size":0,"ondisk_log_size":1219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:10:40.748016+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0012192360000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6865,"num_read_kb":17213,"num_write":1894,"num_write_kb":6549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1502956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"160'407","reported_seq":4782,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.810643+0000","last_change":"2024-09-04T21:56:45.810643+0000","last_active":"2024-09-04T21:56:45.810643+0000","last_peered":"2024-09-04T21:56:45.810643+0000","last_clean":"2024-09-04T21:56:45.810643+0000","last_became_active":"2024-09-04T21:56:05.369067+0000","last_became_peered":"2024-09-04T21:56:05.369067+0000","last_unstale":"2024-09-04T21:56:45.810643+0000","last_undegraded":"2024-09-04T21:56:45.810643+0000","last_fullsized":"2024-09-04T21:56:45.810643+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'407","last_scrub_stamp":"2024-09-04T21:56:45.810455+0000","last_deep_scrub":"160'407","last_deep_scrub_stamp":"2024-09-04T21:56:45.810455+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.810455+0000","objects_scrubbed":5,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:26:22.607912+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0036310650000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5144,"num_read_kb":26186,"num_write":389,"num_write_kb":8125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":9411895,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"160'969","reported_seq":3266,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.151122+0000","last_change":"2024-09-04T21:56:45.151122+0000","last_active":"2024-09-04T21:56:45.151122+0000","last_peered":"2024-09-04T21:56:45.151122+0000","last_clean":"2024-09-04T21:56:45.151122+0000","last_became_active":"2024-09-04T21:56:06.366559+0000","last_became_peered":"2024-09-04T21:56:06.366559+0000","last_unstale":"2024-09-04T21:56:45.151122+0000","last_undegraded":"2024-09-04T21:56:45.151122+0000","last_fullsized":"2024-09-04T21:56:45.151122+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'969","last_scrub_stamp":"2024-09-04T21:56:45.150923+0000","last_deep_scrub":"160'969","last_deep_scrub_stamp":"2024-09-04T21:56:45.150923+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.150923+0000","objects_scrubbed":13,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:06:38.718407+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.001349694,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":5747,"num_read_kb":22435,"num_write":1319,"num_write_kb":7116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":4773566,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"160'364","reported_seq":2723,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:48.062561+0000","last_change":"2024-09-04T21:56:48.062561+0000","last_active":"2024-09-04T21:56:48.062561+0000","last_peered":"2024-09-04T21:56:48.062561+0000","last_clean":"2024-09-04T21:56:48.062561+0000","last_became_active":"2024-09-04T21:56:29.615094+0000","last_became_peered":"2024-09-04T21:56:29.615094+0000","last_unstale":"2024-09-04T21:56:48.062561+0000","last_undegraded":"2024-09-04T21:56:48.062561+0000","last_fullsized":"2024-09-04T21:56:48.062561+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'364","last_scrub_stamp":"2024-09-04T21:56:48.062343+0000","last_deep_scrub":"160'364","last_deep_scrub_stamp":"2024-09-04T21:56:48.062343+0000","last_clean_scrub_stamp":"2024-09-04T21:56:48.062343+0000","objects_scrubbed":7,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T06:23:27.653070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0025116219999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2897,"num_read_kb":16809,"num_write":331,"num_write_kb":6472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1558528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"160'406","reported_seq":3191,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:51.311429+0000","last_change":"2024-09-04T21:56:51.311264+0000","last_active":"2024-09-04T21:56:51.311429+0000","last_peered":"2024-09-04T21:56:51.311429+0000","last_clean":"2024-09-04T21:56:51.311429+0000","last_became_active":"2024-09-04T21:54:31.583560+0000","last_became_peered":"2024-09-04T21:54:31.583560+0000","last_unstale":"2024-09-04T21:56:51.311429+0000","last_undegraded":"2024-09-04T21:56:51.311429+0000","last_fullsized":"2024-09-04T21:56:51.311429+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'406","last_scrub_stamp":"2024-09-04T21:56:51.311053+0000","last_deep_scrub":"160'406","last_deep_scrub_stamp":"2024-09-04T21:56:51.311053+0000","last_clean_scrub_stamp":"2024-09-04T21:56:51.311053+0000","objects_scrubbed":13,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:28:20.524456+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.005116766,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":3204,"num_read_kb":27792,"num_write":394,"num_write_kb":24221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4861244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"160'347","reported_seq":2085,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.616304+0000","last_change":"2024-09-04T21:56:45.616304+0000","last_active":"2024-09-04T21:56:45.616304+0000","last_peered":"2024-09-04T21:56:45.616304+0000","last_clean":"2024-09-04T21:56:45.616304+0000","last_became_active":"2024-09-04T21:55:41.366959+0000","last_became_peered":"2024-09-04T21:55:41.366959+0000","last_unstale":"2024-09-04T21:56:45.616304+0000","last_undegraded":"2024-09-04T21:56:45.616304+0000","last_fullsized":"2024-09-04T21:56:45.616304+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'347","last_scrub_stamp":"2024-09-04T21:56:45.616107+0000","last_deep_scrub":"160'347","last_deep_scrub_stamp":"2024-09-04T21:56:45.616107+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.616107+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:03:22.612282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029305849999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":1596,"num_read_kb":6400,"num_write":363,"num_write_kb":4349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":591326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":313,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.512524+0000","last_change":"2024-09-04T21:56:46.512524+0000","last_active":"2024-09-04T21:56:46.512524+0000","last_peered":"2024-09-04T21:56:46.512524+0000","last_clean":"2024-09-04T21:56:46.512524+0000","last_became_active":"2024-09-04T21:54:31.580805+0000","last_became_peered":"2024-09-04T21:54:31.580805+0000","last_unstale":"2024-09-04T21:56:46.512524+0000","last_undegraded":"2024-09-04T21:56:46.512524+0000","last_fullsized":"2024-09-04T21:56:46.512524+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:46.512347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:46.512347+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.512347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:43:23.792480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012180050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"160'470","reported_seq":2562,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:44.126550+0000","last_change":"2024-09-04T21:56:44.126550+0000","last_active":"2024-09-04T21:56:44.126550+0000","last_peered":"2024-09-04T21:56:44.126550+0000","last_clean":"2024-09-04T21:56:44.126550+0000","last_became_active":"2024-09-04T21:56:04.383844+0000","last_became_peered":"2024-09-04T21:56:04.383844+0000","last_unstale":"2024-09-04T21:56:44.126550+0000","last_undegraded":"2024-09-04T21:56:44.126550+0000","last_fullsized":"2024-09-04T21:56:44.126550+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'470","last_scrub_stamp":"2024-09-04T21:56:44.126482+0000","last_deep_scrub":"160'470","last_deep_scrub_stamp":"2024-09-04T21:56:44.126482+0000","last_clean_scrub_stamp":"2024-09-04T21:56:44.126482+0000","objects_scrubbed":5,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T09:37:21.498891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092818800000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3307,"num_read_kb":5460,"num_write":390,"num_write_kb":7375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":2056615,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":102,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.474646+0000","last_change":"2024-09-04T21:56:43.474646+0000","last_active":"2024-09-04T21:56:43.474646+0000","last_peered":"2024-09-04T21:56:43.474646+0000","last_clean":"2024-09-04T21:56:43.474646+0000","last_became_active":"2024-09-04T21:55:50.372367+0000","last_became_peered":"2024-09-04T21:55:50.372367+0000","last_unstale":"2024-09-04T21:56:43.474646+0000","last_undegraded":"2024-09-04T21:56:43.474646+0000","last_fullsized":"2024-09-04T21:56:43.474646+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:43.474563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:43.474563+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.474563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:24:16.496371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"160'357","reported_seq":3587,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.808342+0000","last_change":"2024-09-04T21:56:43.808342+0000","last_active":"2024-09-04T21:56:43.808342+0000","last_peered":"2024-09-04T21:56:43.808342+0000","last_clean":"2024-09-04T21:56:43.808342+0000","last_became_active":"2024-09-04T21:54:31.589174+0000","last_became_peered":"2024-09-04T21:54:31.589174+0000","last_unstale":"2024-09-04T21:56:43.808342+0000","last_undegraded":"2024-09-04T21:56:43.808342+0000","last_fullsized":"2024-09-04T21:56:43.808342+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'357","last_scrub_stamp":"2024-09-04T21:56:43.808180+0000","last_deep_scrub":"160'357","last_deep_scrub_stamp":"2024-09-04T21:56:43.808180+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.808180+0000","objects_scrubbed":4,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:05:03.226730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0056745370000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":3516,"num_read_kb":22165,"num_write":358,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2402067,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":280,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.280759+0000","last_change":"2024-09-04T21:56:46.280759+0000","last_active":"2024-09-04T21:56:46.280759+0000","last_peered":"2024-09-04T21:56:46.280759+0000","last_clean":"2024-09-04T21:56:46.280759+0000","last_became_active":"2024-09-04T21:56:01.382649+0000","last_became_peered":"2024-09-04T21:56:01.382649+0000","last_unstale":"2024-09-04T21:56:46.280759+0000","last_undegraded":"2024-09-04T21:56:46.280759+0000","last_fullsized":"2024-09-04T21:56:46.280759+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:46.280581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:46.280581+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.280581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T02:29:37.129210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"160'336","reported_seq":1985,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.495176+0000","last_change":"2024-09-04T21:56:45.495176+0000","last_active":"2024-09-04T21:56:45.495176+0000","last_peered":"2024-09-04T21:56:45.495176+0000","last_clean":"2024-09-04T21:56:45.495176+0000","last_became_active":"2024-09-04T21:55:53.479895+0000","last_became_peered":"2024-09-04T21:55:53.479895+0000","last_unstale":"2024-09-04T21:56:45.495176+0000","last_undegraded":"2024-09-04T21:56:45.495176+0000","last_fullsized":"2024-09-04T21:56:45.495176+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'336","last_scrub_stamp":"2024-09-04T21:56:45.494987+0000","last_deep_scrub":"160'336","last_deep_scrub_stamp":"2024-09-04T21:56:45.494987+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.494987+0000","objects_scrubbed":6,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:37:27.317420+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00092870599999999995,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":1989,"num_read_kb":8013,"num_write":285,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":1256925,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":108,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.506849+0000","last_change":"2024-09-04T21:56:42.506849+0000","last_active":"2024-09-04T21:56:42.506849+0000","last_peered":"2024-09-04T21:56:42.506849+0000","last_clean":"2024-09-04T21:56:42.506849+0000","last_became_active":"2024-09-04T21:56:01.380215+0000","last_became_peered":"2024-09-04T21:56:01.380215+0000","last_unstale":"2024-09-04T21:56:42.506849+0000","last_undegraded":"2024-09-04T21:56:42.506849+0000","last_fullsized":"2024-09-04T21:56:42.506849+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:42.506758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:42.506758+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.506758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:26:15.610554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"160'566","reported_seq":11932,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.146131+0000","last_change":"2024-09-04T21:56:46.146131+0000","last_active":"2024-09-04T21:56:46.146131+0000","last_peered":"2024-09-04T21:56:46.146131+0000","last_clean":"2024-09-04T21:56:46.146131+0000","last_became_active":"2024-09-04T21:56:29.610534+0000","last_became_peered":"2024-09-04T21:56:29.610534+0000","last_unstale":"2024-09-04T21:56:46.146131+0000","last_undegraded":"2024-09-04T21:56:46.146131+0000","last_fullsized":"2024-09-04T21:56:46.146131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'566","last_scrub_stamp":"2024-09-04T21:56:46.145843+0000","last_deep_scrub":"160'566","last_deep_scrub_stamp":"2024-09-04T21:56:46.145843+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.145843+0000","objects_scrubbed":6,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T23:00:51.127444+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0085876379999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":12771,"num_read_kb":25430,"num_write":599,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"160'514","reported_seq":4134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.592525+0000","last_change":"2024-09-04T21:56:43.592525+0000","last_active":"2024-09-04T21:56:43.592525+0000","last_peered":"2024-09-04T21:56:43.592525+0000","last_clean":"2024-09-04T21:56:43.592525+0000","last_became_active":"2024-09-04T21:56:05.372788+0000","last_became_peered":"2024-09-04T21:56:05.372788+0000","last_unstale":"2024-09-04T21:56:43.592525+0000","last_undegraded":"2024-09-04T21:56:43.592525+0000","last_fullsized":"2024-09-04T21:56:43.592525+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'514","last_scrub_stamp":"2024-09-04T21:56:43.592272+0000","last_deep_scrub":"160'514","last_deep_scrub_stamp":"2024-09-04T21:56:43.592272+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.592272+0000","objects_scrubbed":6,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:13:46.724484+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0055132489999999996,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":3371,"num_read_kb":16341,"num_write":551,"num_write_kb":12408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":2227387,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"160'3","reported_seq":135,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:47.092180+0000","last_change":"2024-09-04T21:56:47.092180+0000","last_active":"2024-09-04T21:56:47.092180+0000","last_peered":"2024-09-04T21:56:47.092180+0000","last_clean":"2024-09-04T21:56:47.092180+0000","last_became_active":"2024-09-04T21:56:29.609626+0000","last_became_peered":"2024-09-04T21:56:29.609626+0000","last_unstale":"2024-09-04T21:56:47.092180+0000","last_undegraded":"2024-09-04T21:56:47.092180+0000","last_fullsized":"2024-09-04T21:56:47.092180+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'3","last_scrub_stamp":"2024-09-04T21:56:47.092028+0000","last_deep_scrub":"160'3","last_deep_scrub_stamp":"2024-09-04T21:56:47.092028+0000","last_clean_scrub_stamp":"2024-09-04T21:56:47.092028+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:33:32.895966+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0022353749999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"160'384","reported_seq":3321,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:46.467364+0000","last_change":"2024-09-04T21:56:46.467364+0000","last_active":"2024-09-04T21:56:46.467364+0000","last_peered":"2024-09-04T21:56:46.467364+0000","last_clean":"2024-09-04T21:56:46.467364+0000","last_became_active":"2024-09-04T21:55:53.479749+0000","last_became_peered":"2024-09-04T21:55:53.479749+0000","last_unstale":"2024-09-04T21:56:46.467364+0000","last_undegraded":"2024-09-04T21:56:46.467364+0000","last_fullsized":"2024-09-04T21:56:46.467364+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'384","last_scrub_stamp":"2024-09-04T21:56:46.467171+0000","last_deep_scrub":"160'384","last_deep_scrub_stamp":"2024-09-04T21:56:46.467171+0000","last_clean_scrub_stamp":"2024-09-04T21:56:46.467171+0000","objects_scrubbed":8,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:05:22.650833+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.001047089,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3750,"num_read_kb":53898,"num_write":586,"num_write_kb":21349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":3098752,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.714491+0000","last_change":"2024-09-04T21:56:42.714491+0000","last_active":"2024-09-04T21:56:42.714491+0000","last_peered":"2024-09-04T21:56:42.714491+0000","last_clean":"2024-09-04T21:56:42.714491+0000","last_became_active":"2024-09-04T21:53:45.789901+0000","last_became_peered":"2024-09-04T21:53:45.789901+0000","last_unstale":"2024-09-04T21:56:42.714491+0000","last_undegraded":"2024-09-04T21:56:42.714491+0000","last_fullsized":"2024-09-04T21:56:42.714491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:42.714384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:42.714384+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.714384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:34:18.287606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010674009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'625","reported_seq":4706,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.107231+0000","last_change":"2024-09-04T21:56:42.107231+0000","last_active":"2024-09-04T21:56:42.107231+0000","last_peered":"2024-09-04T21:56:42.107231+0000","last_clean":"2024-09-04T21:56:42.107231+0000","last_became_active":"2024-09-04T21:56:04.387890+0000","last_became_peered":"2024-09-04T21:56:04.387890+0000","last_unstale":"2024-09-04T21:56:42.107231+0000","last_undegraded":"2024-09-04T21:56:42.107231+0000","last_fullsized":"2024-09-04T21:56:42.107231+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'625","last_scrub_stamp":"2024-09-04T21:56:42.107144+0000","last_deep_scrub":"160'625","last_deep_scrub_stamp":"2024-09-04T21:56:42.107144+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.107144+0000","objects_scrubbed":7,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:22:32.286933+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00076480899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":6211,"num_read_kb":34278,"num_write":864,"num_write_kb":14494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":6457180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":268,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:49.281860+0000","last_change":"2024-09-04T21:56:49.281860+0000","last_active":"2024-09-04T21:56:49.281860+0000","last_peered":"2024-09-04T21:56:49.281860+0000","last_clean":"2024-09-04T21:56:49.281860+0000","last_became_active":"2024-09-04T21:55:50.372748+0000","last_became_peered":"2024-09-04T21:55:50.372748+0000","last_unstale":"2024-09-04T21:56:49.281860+0000","last_undegraded":"2024-09-04T21:56:49.281860+0000","last_fullsized":"2024-09-04T21:56:49.281860+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-04T21:56:49.281673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-04T21:56:49.281673+0000","last_clean_scrub_stamp":"2024-09-04T21:56:49.281673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:54:27.144115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"160'438","reported_seq":3668,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:42.863035+0000","last_change":"2024-09-04T21:56:42.863035+0000","last_active":"2024-09-04T21:56:42.863035+0000","last_peered":"2024-09-04T21:56:42.863035+0000","last_clean":"2024-09-04T21:56:42.863035+0000","last_became_active":"2024-09-04T21:53:55.344474+0000","last_became_peered":"2024-09-04T21:53:55.344474+0000","last_unstale":"2024-09-04T21:56:42.863035+0000","last_undegraded":"2024-09-04T21:56:42.863035+0000","last_fullsized":"2024-09-04T21:56:42.863035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'438","last_scrub_stamp":"2024-09-04T21:56:42.862938+0000","last_deep_scrub":"160'438","last_deep_scrub_stamp":"2024-09-04T21:56:42.862938+0000","last_clean_scrub_stamp":"2024-09-04T21:56:42.862938+0000","objects_scrubbed":12,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:21:20.880609+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0035691910000000002,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":3941,"num_read_kb":15042,"num_write":381,"num_write_kb":5849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"160'2","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:44.192397+0000","last_change":"2024-09-04T21:56:44.192397+0000","last_active":"2024-09-04T21:56:44.192397+0000","last_peered":"2024-09-04T21:56:44.192397+0000","last_clean":"2024-09-04T21:56:44.192397+0000","last_became_active":"2024-09-04T21:56:29.612875+0000","last_became_peered":"2024-09-04T21:56:29.612875+0000","last_unstale":"2024-09-04T21:56:44.192397+0000","last_undegraded":"2024-09-04T21:56:44.192397+0000","last_fullsized":"2024-09-04T21:56:44.192397+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'2","last_scrub_stamp":"2024-09-04T21:56:44.192218+0000","last_deep_scrub":"160'2","last_deep_scrub_stamp":"2024-09-04T21:56:44.192218+0000","last_clean_scrub_stamp":"2024-09-04T21:56:44.192218+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T00:43:19.898637+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0012332389999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"160'74","reported_seq":366,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.323455+0000","last_change":"2024-09-04T21:56:45.323455+0000","last_active":"2024-09-04T21:56:45.323455+0000","last_peered":"2024-09-04T21:56:45.323455+0000","last_clean":"2024-09-04T21:56:45.323455+0000","last_became_active":"2024-09-04T21:56:01.382135+0000","last_became_peered":"2024-09-04T21:56:01.382135+0000","last_unstale":"2024-09-04T21:56:45.323455+0000","last_undegraded":"2024-09-04T21:56:45.323455+0000","last_fullsized":"2024-09-04T21:56:45.323455+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'74","last_scrub_stamp":"2024-09-04T21:56:45.323236+0000","last_deep_scrub":"160'74","last_deep_scrub_stamp":"2024-09-04T21:56:45.323236+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.323236+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:36:17.874217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"160'597","reported_seq":3835,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:44.803496+0000","last_change":"2024-09-04T21:56:44.803496+0000","last_active":"2024-09-04T21:56:44.803496+0000","last_peered":"2024-09-04T21:56:44.803496+0000","last_clean":"2024-09-04T21:56:44.803496+0000","last_became_active":"2024-09-04T21:55:54.486363+0000","last_became_peered":"2024-09-04T21:55:54.486363+0000","last_unstale":"2024-09-04T21:56:44.803496+0000","last_undegraded":"2024-09-04T21:56:44.803496+0000","last_fullsized":"2024-09-04T21:56:44.803496+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'597","last_scrub_stamp":"2024-09-04T21:56:44.803424+0000","last_deep_scrub":"160'597","last_deep_scrub_stamp":"2024-09-04T21:56:44.803424+0000","last_clean_scrub_stamp":"2024-09-04T21:56:44.803424+0000","objects_scrubbed":7,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:51:11.265976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0055292229999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4421,"num_read_kb":15178,"num_write":685,"num_write_kb":7069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":5861572,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"160'350","reported_seq":2216,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:44.637606+0000","last_change":"2024-09-04T21:56:44.637606+0000","last_active":"2024-09-04T21:56:44.637606+0000","last_peered":"2024-09-04T21:56:44.637606+0000","last_clean":"2024-09-04T21:56:44.637606+0000","last_became_active":"2024-09-04T21:55:38.753242+0000","last_became_peered":"2024-09-04T21:55:38.753242+0000","last_unstale":"2024-09-04T21:56:44.637606+0000","last_undegraded":"2024-09-04T21:56:44.637606+0000","last_fullsized":"2024-09-04T21:56:44.637606+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'350","last_scrub_stamp":"2024-09-04T21:56:44.637409+0000","last_deep_scrub":"160'350","last_deep_scrub_stamp":"2024-09-04T21:56:44.637409+0000","last_clean_scrub_stamp":"2024-09-04T21:56:44.637409+0000","objects_scrubbed":5,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-05T22:17:54.344504+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0058233,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":2780,"num_read_kb":16422,"num_write":306,"num_write_kb":6619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":105355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"160'710","reported_seq":2989,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:44.517385+0000","last_change":"2024-09-04T21:56:44.517385+0000","last_active":"2024-09-04T21:56:44.517385+0000","last_peered":"2024-09-04T21:56:44.517385+0000","last_clean":"2024-09-04T21:56:44.517385+0000","last_became_active":"2024-09-04T21:55:41.362544+0000","last_became_peered":"2024-09-04T21:55:41.362544+0000","last_unstale":"2024-09-04T21:56:44.517385+0000","last_undegraded":"2024-09-04T21:56:44.517385+0000","last_fullsized":"2024-09-04T21:56:44.517385+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'710","last_scrub_stamp":"2024-09-04T21:56:44.517191+0000","last_deep_scrub":"160'710","last_deep_scrub_stamp":"2024-09-04T21:56:44.517191+0000","last_clean_scrub_stamp":"2024-09-04T21:56:44.517191+0000","objects_scrubbed":5,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:01:46.884496+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0014608399999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":3930,"num_read_kb":13621,"num_write":826,"num_write_kb":9502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":50228,"num_keys_recovered":18,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"160'390","reported_seq":2472,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:47.829330+0000","last_change":"2024-09-04T21:56:47.829330+0000","last_active":"2024-09-04T21:56:47.829330+0000","last_peered":"2024-09-04T21:56:47.829330+0000","last_clean":"2024-09-04T21:56:47.829330+0000","last_became_active":"2024-09-04T21:56:06.365135+0000","last_became_peered":"2024-09-04T21:56:06.365135+0000","last_unstale":"2024-09-04T21:56:47.829330+0000","last_undegraded":"2024-09-04T21:56:47.829330+0000","last_fullsized":"2024-09-04T21:56:47.829330+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'390","last_scrub_stamp":"2024-09-04T21:56:47.829134+0000","last_deep_scrub":"160'390","last_deep_scrub_stamp":"2024-09-04T21:56:47.829134+0000","last_clean_scrub_stamp":"2024-09-04T21:56:47.829134+0000","objects_scrubbed":7,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:02:48.253450+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0043040520000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":2921,"num_read_kb":7890,"num_write":429,"num_write_kb":6209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":3611800,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"160'363","reported_seq":3611,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.175615+0000","last_change":"2024-09-04T21:56:45.175615+0000","last_active":"2024-09-04T21:56:45.175615+0000","last_peered":"2024-09-04T21:56:45.175615+0000","last_clean":"2024-09-04T21:56:45.175615+0000","last_became_active":"2024-09-04T21:56:29.610180+0000","last_became_peered":"2024-09-04T21:56:29.610180+0000","last_unstale":"2024-09-04T21:56:45.175615+0000","last_undegraded":"2024-09-04T21:56:45.175615+0000","last_fullsized":"2024-09-04T21:56:45.175615+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'363","last_scrub_stamp":"2024-09-04T21:56:45.175392+0000","last_deep_scrub":"160'363","last_deep_scrub_stamp":"2024-09-04T21:56:45.175392+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.175392+0000","objects_scrubbed":6,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T08:39:52.954158+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0048801180000000001,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3701,"num_read_kb":12906,"num_write":315,"num_write_kb":6084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"160'429","reported_seq":11236,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:47.347080+0000","last_change":"2024-09-04T21:56:47.346824+0000","last_active":"2024-09-04T21:56:47.347080+0000","last_peered":"2024-09-04T21:56:47.347080+0000","last_clean":"2024-09-04T21:56:47.347080+0000","last_became_active":"2024-09-04T21:54:31.580029+0000","last_became_peered":"2024-09-04T21:54:31.580029+0000","last_unstale":"2024-09-04T21:56:47.347080+0000","last_undegraded":"2024-09-04T21:56:47.347080+0000","last_fullsized":"2024-09-04T21:56:47.347080+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'429","last_scrub_stamp":"2024-09-04T21:56:47.346479+0000","last_deep_scrub":"160'429","last_deep_scrub_stamp":"2024-09-04T21:56:47.346479+0000","last_clean_scrub_stamp":"2024-09-04T21:56:47.346479+0000","objects_scrubbed":10,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:25:02.003792+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0057126870000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":11855,"num_read_kb":18507,"num_write":393,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2836845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"160'400","reported_seq":1403,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:48.314777+0000","last_change":"2024-09-04T21:56:48.314612+0000","last_active":"2024-09-04T21:56:48.314777+0000","last_peered":"2024-09-04T21:56:48.314777+0000","last_clean":"2024-09-04T21:56:48.314777+0000","last_became_active":"2024-09-04T21:55:41.368964+0000","last_became_peered":"2024-09-04T21:55:41.368964+0000","last_unstale":"2024-09-04T21:56:48.314777+0000","last_undegraded":"2024-09-04T21:56:48.314777+0000","last_fullsized":"2024-09-04T21:56:48.314777+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'400","last_scrub_stamp":"2024-09-04T21:56:48.314416+0000","last_deep_scrub":"160'400","last_deep_scrub_stamp":"2024-09-04T21:56:48.314416+0000","last_clean_scrub_stamp":"2024-09-04T21:56:48.314416+0000","objects_scrubbed":6,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T04:23:55.581521+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0050823930000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2160,"num_read_kb":12559,"num_write":460,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":914460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"160'499","reported_seq":11970,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:50.275643+0000","last_change":"2024-09-04T21:56:50.275643+0000","last_active":"2024-09-04T21:56:50.275643+0000","last_peered":"2024-09-04T21:56:50.275643+0000","last_clean":"2024-09-04T21:56:50.275643+0000","last_became_active":"2024-09-04T21:55:38.771576+0000","last_became_peered":"2024-09-04T21:55:38.771576+0000","last_unstale":"2024-09-04T21:56:50.275643+0000","last_undegraded":"2024-09-04T21:56:50.275643+0000","last_fullsized":"2024-09-04T21:56:50.275643+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'499","last_scrub_stamp":"2024-09-04T21:56:50.275381+0000","last_deep_scrub":"160'499","last_deep_scrub_stamp":"2024-09-04T21:56:50.275381+0000","last_clean_scrub_stamp":"2024-09-04T21:56:50.275381+0000","objects_scrubbed":11,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T07:57:20.285058+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0054674930000000004,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":11629,"num_read_kb":24785,"num_write":466,"num_write_kb":10770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":851952,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"160'839","reported_seq":3726,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:45.501880+0000","last_change":"2024-09-04T21:56:45.501880+0000","last_active":"2024-09-04T21:56:45.501880+0000","last_peered":"2024-09-04T21:56:45.501880+0000","last_clean":"2024-09-04T21:56:45.501880+0000","last_became_active":"2024-09-04T21:56:05.373439+0000","last_became_peered":"2024-09-04T21:56:05.373439+0000","last_unstale":"2024-09-04T21:56:45.501880+0000","last_undegraded":"2024-09-04T21:56:45.501880+0000","last_fullsized":"2024-09-04T21:56:45.501880+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'839","last_scrub_stamp":"2024-09-04T21:56:45.501687+0000","last_deep_scrub":"160'839","last_deep_scrub_stamp":"2024-09-04T21:56:45.501687+0000","last_clean_scrub_stamp":"2024-09-04T21:56:45.501687+0000","objects_scrubbed":9,"log_size":839,"log_dups_size":0,"ondisk_log_size":839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T05:32:07.767021+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.001333475,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5332,"num_read_kb":12792,"num_write":1029,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":5041623,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"160'584","reported_seq":6078,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:52.284632+0000","last_change":"2024-09-04T21:56:52.284476+0000","last_active":"2024-09-04T21:56:52.284632+0000","last_peered":"2024-09-04T21:56:52.284632+0000","last_clean":"2024-09-04T21:56:52.284632+0000","last_became_active":"2024-09-04T21:55:38.766350+0000","last_became_peered":"2024-09-04T21:55:38.766350+0000","last_unstale":"2024-09-04T21:56:52.284632+0000","last_undegraded":"2024-09-04T21:56:52.284632+0000","last_fullsized":"2024-09-04T21:56:52.284632+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'584","last_scrub_stamp":"2024-09-04T21:56:52.284282+0000","last_deep_scrub":"160'584","last_deep_scrub_stamp":"2024-09-04T21:56:52.284282+0000","last_clean_scrub_stamp":"2024-09-04T21:56:52.284282+0000","objects_scrubbed":5,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:16:06.334134+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0044194070000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":6859,"num_read_kb":44210,"num_write":682,"num_write_kb":25337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6771278,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"160'768","reported_seq":13648,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:43.101463+0000","last_change":"2024-09-04T21:56:43.101463+0000","last_active":"2024-09-04T21:56:43.101463+0000","last_peered":"2024-09-04T21:56:43.101463+0000","last_clean":"2024-09-04T21:56:43.101463+0000","last_became_active":"2024-09-04T21:56:05.373906+0000","last_became_peered":"2024-09-04T21:56:05.373906+0000","last_unstale":"2024-09-04T21:56:43.101463+0000","last_undegraded":"2024-09-04T21:56:43.101463+0000","last_fullsized":"2024-09-04T21:56:43.101463+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'768","last_scrub_stamp":"2024-09-04T21:56:43.101381+0000","last_deep_scrub":"160'768","last_deep_scrub_stamp":"2024-09-04T21:56:43.101381+0000","last_clean_scrub_stamp":"2024-09-04T21:56:43.101381+0000","objects_scrubbed":5,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T03:18:07.772307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020518289999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":14612,"num_read_kb":38088,"num_write":913,"num_write_kb":9473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":9415549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"160'980","reported_seq":2660,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-04T21:56:53.329027+0000","last_change":"2024-09-04T21:56:53.328878+0000","last_active":"2024-09-04T21:56:53.329027+0000","last_peered":"2024-09-04T21:56:53.329027+0000","last_clean":"2024-09-04T21:56:53.329027+0000","last_became_active":"2024-09-04T21:55:41.368871+0000","last_became_peered":"2024-09-04T21:55:41.368871+0000","last_unstale":"2024-09-04T21:56:53.329027+0000","last_undegraded":"2024-09-04T21:56:53.329027+0000","last_fullsized":"2024-09-04T21:56:53.329027+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'980","last_scrub_stamp":"2024-09-04T21:56:53.328678+0000","last_deep_scrub":"160'980","last_deep_scrub_stamp":"2024-09-04T21:56:53.328678+0000","last_clean_scrub_stamp":"2024-09-04T21:56:53.328678+0000","objects_scrubbed":7,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-06T01:22:08.372568+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0021491650000000002,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4705,"num_read_kb":19308,"num_write":1473,"num_write_kb":7294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1637823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":225,"num_read":192313,"num_read_kb":673893,"num_write":20722,"num_write_kb":314413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":89803858,"num_keys_recovered":307,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},"log_size":17710,"ondisk_log_size":17710,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":19394,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509486,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":105,"seq":450971566099,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39268,"kb_used_data":1876,"kb_used_omap":33,"kb_used_meta":37342,"kb_avail":94332572,"statfs":{"total":96636764160,"available":96596553728,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34021,"internal_metadata":38239003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509486,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":919578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509487,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51144,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320696,"statfs":{"total":96636764160,"available":96584392704,"internally_reserved":0,"allocated":1957888,"data_stored":1509908,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542189,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59424,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312416,"statfs":{"total":96636764160,"available":96575913984,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542190,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45536,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326304,"statfs":{"total":96636764160,"available":96590135296,"internally_reserved":0,"allocated":1916928,"data_stored":919540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430240,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34072,"kb_used_data":1928,"kb_used_omap":4,"kb_used_meta":32123,"kb_avail":94337768,"statfs":{"total":96636764160,"available":96601874432,"internally_reserved":0,"allocated":1974272,"data_stored":1509908,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":4394,"internal_metadata":32894678},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59428,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312412,"statfs":{"total":96636764160,"available":96575909888,"internally_reserved":0,"allocated":1921024,"data_stored":919559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6735,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5924,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-04T21:57:05.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-04T21:57:05.775 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-04T21:57:05.775 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-04T21:57:05.775 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-04T21:57:05.775 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:05.860 INFO:tasks.ceph.osd.0:Stopped 2024-09-04T21:57:05.860 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-04T21:57:05.860 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:05.938 INFO:tasks.ceph.osd.1:Stopped 2024-09-04T21:57:05.939 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-04T21:57:05.939 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:06.024 INFO:tasks.ceph.osd.2:Stopped 2024-09-04T21:57:06.025 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-04T21:57:06.025 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:06.101 INFO:tasks.ceph.osd.3:Stopped 2024-09-04T21:57:06.102 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-04T21:57:06.102 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:11.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-04T21:57:12.204 INFO:tasks.ceph.osd.4:Stopped 2024-09-04T21:57:12.205 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-04T21:57:12.205 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.276 INFO:tasks.ceph.osd.5:Stopped 2024-09-04T21:57:12.276 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-04T21:57:12.276 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.358 INFO:tasks.ceph.osd.6:Stopped 2024-09-04T21:57:12.358 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-04T21:57:12.358 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.443 INFO:tasks.ceph.osd.7:Stopped 2024-09-04T21:57:12.443 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-04T21:57:12.444 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-04T21:57:12.444 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.473 INFO:tasks.ceph.mgr.y:Stopped 2024-09-04T21:57:12.474 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-04T21:57:12.474 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.502 INFO:tasks.ceph.mgr.x:Stopped 2024-09-04T21:57:12.502 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-04T21:57:12.502 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-04T21:57:12.503 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.530 INFO:tasks.ceph.mon.a:Stopped 2024-09-04T21:57:12.531 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-04T21:57:12.531 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.590 INFO:tasks.ceph.mon.c:Stopped 2024-09-04T21:57:12.590 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-04T21:57:12.590 INFO:teuthology.orchestra.run:waiting for 300 2024-09-04T21:57:12.615 INFO:tasks.ceph.mon.b:Stopped 2024-09-04T21:57:12.615 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-04T21:57:12.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-04T21:57:12.650 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:57:12.650 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-04T21:57:13.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:57:13.029 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-04T21:57:13.253 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:57:13.253 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-04T21:57:13.455 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:57:13.455 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-04T21:57:13.647 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:57:13.647 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-04T21:57:13.869 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:57:13.870 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-04T21:57:14.085 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:57:14.085 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-04T21:57:14.276 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:57:14.276 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-04T21:57:14.444 INFO:tasks.ceph:Archiving mon data... 2024-09-04T21:57:14.445 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/data/mon.a.tgz 2024-09-04T21:57:14.445 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-04T21:57:14.451 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.AQqRKtGtXL 2024-09-04T21:57:14.452 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.AQqRKtGtXL 2024-09-04T21:57:14.597 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.AQqRKtGtXL 2024-09-04T21:57:14.662 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.AQqRKtGtXL is 417KB 2024-09-04T21:57:14.724 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.AQqRKtGtXL 2024-09-04T21:57:14.729 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/data/mon.c.tgz 2024-09-04T21:57:14.729 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-04T21:57:14.778 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.XDpYR6gJIw 2024-09-04T21:57:14.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.XDpYR6gJIw 2024-09-04T21:57:14.960 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.XDpYR6gJIw 2024-09-04T21:57:15.022 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.XDpYR6gJIw is 572KB 2024-09-04T21:57:15.088 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.XDpYR6gJIw 2024-09-04T21:57:15.094 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/data/mon.b.tgz 2024-09-04T21:57:15.094 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-09-04T21:57:15.100 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.91LlYsp3ED 2024-09-04T21:57:15.100 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.91LlYsp3ED 2024-09-04T21:57:15.278 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.91LlYsp3ED 2024-09-04T21:57:15.338 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.91LlYsp3ED is 574KB 2024-09-04T21:57:15.401 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.91LlYsp3ED 2024-09-04T21:57:15.406 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-04T21:57:15.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-04T21:57:15.409 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-04T21:57:15.466 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-04T21:57:15.504 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-04T21:57:15.504 INFO:tasks.ceph:Archiving crash dumps... 2024-09-04T21:57:15.506 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/remote/smithi112/crash 2024-09-04T21:57:15.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-04T21:57:15.526 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/remote/smithi157/crash 2024-09-04T21:57:15.526 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-04T21:57:15.560 INFO:tasks.ceph:Compressing logs... 2024-09-04T21:57:15.560 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-04T21:57:15.567 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-04T21:57:15.591 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30957.log 2024-09-04T21:57:15.591 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36922.log 2024-09-04T21:57:15.592 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30957.log.gz 2024-09-04T21:57:15.592 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31035.log 2024-09-04T21:57:15.592 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36922.log.gz 2024-09-04T21:57:15.593 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31035.log.gz 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33133.log 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38804.log 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39220.log 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33133.log.gz 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38804.log.gz 2024-09-04T21:57:15.594 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38616.log 2024-09-04T21:57:15.596 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30567.log 2024-09-04T21:57:15.596 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-09-04T21:57:15.596 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38616.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38616.log.gz 2024-09-04T21:57:15.596 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33507.log 2024-09-04T21:57:15.596 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34747.log 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30567.log: /var/log/ceph/ceph-client.admin.33507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33507.log.gz 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30567.log.gz 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31143.log 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34747.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34747.log.gz 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37687.log 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31143.log.gz 2024-09-04T21:57:15.598 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36983.log 2024-09-04T21:57:15.599 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30281.log 2024-09-04T21:57:15.599 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37687.log.gz 2024-09-04T21:57:15.599 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30360.log 2024-09-04T21:57:15.599 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36983.log.gz 2024-09-04T21:57:15.601 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30255.log 2024-09-04T21:57:15.601 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30281.log.gz 2024-09-04T21:57:15.601 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36124.log 2024-09-04T21:57:15.601 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30360.log.gz 2024-09-04T21:57:15.601 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35188.log 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30255.log.gz 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36124.log.gz 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35334.log 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35188.log.gz 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33209.log 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35334.log.gz 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34409.log 2024-09-04T21:57:15.602 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-09-04T21:57:15.603 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30347.log 2024-09-04T21:57:15.603 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33209.log.gz 2024-09-04T21:57:15.608 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38416.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34409.log.gz 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30347.log.gz 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30558.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34495.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38416.log.gz 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39246.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30558.log.gz 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35429.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34495.log.gz 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30983.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39246.log.gz 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32305.log 2024-09-04T21:57:15.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35429.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30983.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30983.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38624.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32305.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39584.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.38624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30549.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38624.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31427.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39584.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30549.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33358.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34270.log 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31427.log.gz 2024-09-04T21:57:15.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33358.log.gz 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34608.log 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39376.log 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34270.log.gz 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34608.log.gz 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39116.log 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2024-09-04T21:57:15.611 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39376.log.gz 2024-09-04T21:57:15.620 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39116.log.gz 2024-09-04T21:57:15.620 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32212.log 2024-09-04T21:57:15.620 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26279.log 2024-09-04T21:57:15.620 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32212.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34296.log 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33691.log 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26279.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34296.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30787.log 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34300.log 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33691.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30787.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38336.log 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34300.log: -- /var/log/ceph/ceph-client.admin.32410.log 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34300.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38336.log.gz 2024-09-04T21:57:15.621 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35309.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32792.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32410.log.gz 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38623.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35309.log.gz 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38284.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32792.log.gz 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38816.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38623.log.gz 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35285.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38284.log.gz 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38816.log.gz 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38403.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31061.log 2024-09-04T21:57:15.622 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35285.log.gz 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37597.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38403.log.gz 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37862.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31061.log.gz 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37597.log.gz 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38605.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37862.log.gz 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38076.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36411.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38605.log.gz 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38076.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38076.log.gz -5 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31009.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-04T21:57:15.623 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36411.log.gz 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31009.log.gz 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38414.log 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38414.log.gz 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30772.log 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37921.log 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37452.log: /var/log/ceph/ceph-client.admin.30772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30772.log.gz 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31904.log 2024-09-04T21:57:15.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30749.log 2024-09-04T21:57:15.625 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37921.log: /var/log/ceph/ceph-client.admin.31904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37921.log.gz 2024-09-04T21:57:15.625 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31904.log.gz 2024-09-04T21:57:15.633 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36011.log 2024-09-04T21:57:15.633 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-09-04T21:57:15.633 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30749.log.gz 2024-09-04T21:57:15.633 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36011.log.gz 2024-09-04T21:57:15.633 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39142.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34899.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35156.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39142.log.gz 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34899.log.gz 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38824.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35156.log.gz 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38824.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.38824.log.gz /var/log/ceph/ceph-client.admin.39038.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36252.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35051.log 2024-09-04T21:57:15.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36252.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38180.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35051.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34497.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38180.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33086.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34497.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36660.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33086.log.gz 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31857.log 2024-09-04T21:57:15.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36660.log.gz 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32661.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34582.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36619.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32661.log.gz 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30355.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34582.log.gz 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36619.log.gz 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33483.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37537.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30355.log: /var/log/ceph/ceph-client.admin.33483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35192.log 2024-09-04T21:57:15.636 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30355.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33483.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38609.log 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37537.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37658.log 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35192.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30569.log 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38609.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34455.log 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31676.log 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30569.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-09-04T21:57:15.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36196.log 2024-09-04T21:57:15.638 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31676.log.gz 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26418.log 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26346.log 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26418.log: gzip -5 --verbose -- 0.0% /var/log/ceph/objectstore_tool.30316.log 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26418.log.gz 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26346.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.33975.log 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26346.log.gz 2024-09-04T21:57:15.659 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.30316.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.33975.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26466.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30512.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr: 94.5% -- replaced with /var/log/ceph/objectstore_tool.30316.log.gz 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26394.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26466.log.gz 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.30512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26322.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26394.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26394.log.gz 2024-09-04T21:57:15.660 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26370.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26322.log.gz 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26298.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26442.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26370.log.gz 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26298.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26298.log.gz 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.33778.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.26442.log: 0.0%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.admin.26442.log.gzgzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.33778.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.log: 95.1% -- replaced with /var/log/ceph/objectstore_tool.33975.log.gz 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr: 94.5% -- replaced with /var/log/ceph/objectstore_tool.33778.log.gzgzip -5 --verbose -- /var/log/ceph/objectstore_tool.31288.log 2024-09-04T21:57:15.661 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.33582.log 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr: 95.2% -- replaced with /var/log/ceph/objectstore_tool.30512.log.gz 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr: 85.1% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/objectstore_tool.31288.log: 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30112.log 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.33582.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.30112.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.b.log: 94.4% -- replaced with /var/log/ceph/objectstore_tool.33582.log.gz 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.34754.log 2024-09-04T21:57:15.662 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.4.log: 94.8% -- replaced with /var/log/ceph/objectstore_tool.30112.log.gz 2024-09-04T21:57:15.663 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/objectstore_tool.34754.log: 95.2% -- replaced with /var/log/ceph/objectstore_tool.31288.log.gz 2024-09-04T21:57:15.664 INFO:teuthology.orchestra.run.smithi157.stderr: 95.1% -- replaced with /var/log/ceph/objectstore_tool.34754.log.gz 2024-09-04T21:57:15.681 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-04T21:57:15.681 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-09-04T21:57:15.681 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36196.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36569.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32977.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37191.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36569.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32149.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32977.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32867.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37191.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33449.log 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32149.log.gz 2024-09-04T21:57:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32867.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32867.log.gz -5 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38407.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37893.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33449.log.gz 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31185.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38407.log.gz 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35542.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37893.log.gz 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31385.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31185.log.gz 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36248.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35542.log.gz 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36461.log 2024-09-04T21:57:15.683 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31385.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36248.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.30557.log -- replaced with /var/log/ceph/ceph-client.admin.36248.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35957.log 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36461.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30557.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26522.log 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35957.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38310.log 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35600.log 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26522.log.gz 2024-09-04T21:57:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38412.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38310.log.gz 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35600.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35600.log.gz 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37131.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38412.log.gz 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30561.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.37131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37131.log.gz 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33082.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30561.log.gz 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36015.log 2024-09-04T21:57:15.685 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33082.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36087.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33665.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30773.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36015.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34843.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33665.log: /var/log/ceph/ceph-client.admin.30773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33589.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33665.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30773.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34843.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38415.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34843.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33589.log.gz 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36854.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35932.log 2024-09-04T21:57:15.686 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38415.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31603.log 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36854.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37506.log 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35932.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30358.log 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31603.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37506.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32258.log 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36339.log 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30358.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30351.log 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32258.log.gz 2024-09-04T21:57:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32592.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36339.log.gz 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30354.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30351.log.gz 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33804.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32592.log.gz 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37243.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30354.log.gz 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33804.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33804.log.gz 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37243.log.gz 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38413.log 2024-09-04T21:57:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30771.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gz 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38413.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38413.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30771.log.gz 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36520.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30767.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36520.log.gz 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31789.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31308.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30767.log.gz 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31789.log.gz 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2024-09-04T21:57:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34059.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31308.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35401.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34059.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21485.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35401.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.tmp-client.admin.21485.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21485.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36375.log 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-09-04T21:57:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31181.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36375.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.38842.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36375.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31181.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31607.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35861.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38842.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37837.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31607.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39064.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35861.log.gz 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32822.log 2024-09-04T21:57:15.691 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37837.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31175.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39064.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33721.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32822.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33015.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.0.31175.log: /var/log/ceph/ceph-client.admin.33721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33390.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33721.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr: 95.3%/var/log/ceph/ceph-client.admin.33015.log: -- replaced with /var/log/ceph/ceph-client.0.31175.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33015.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33162.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33390.log.gz 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30556.log 2024-09-04T21:57:15.692 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33162.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31727.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30556.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32342.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31087.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31727.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34705.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32342.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31087.log.gz 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37567.log 2024-09-04T21:57:15.693 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34705.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35752.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37763.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37567.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35752.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35752.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32068.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37763.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31113.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37763.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32068.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39272.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31113.log.gz 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34650.log 2024-09-04T21:57:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33275.log 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39272.log.gz 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34650.log.gz 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37089.log 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30361.log 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33275.log.gz 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32115.log 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37089.log.gz 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37456.log 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30361.log.gz 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32115.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.32115.log.gz --verbose 2024-09-04T21:57:15.695 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/ceph-client.admin.39090.log 2024-09-04T21:57:15.712 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39506.log 2024-09-04T21:57:15.712 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37456.log: /var/log/ceph/ceph-client.admin.39090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37456.log.gz 2024-09-04T21:57:15.712 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30072.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39506.log: /var/log/ceph/ceph-client.admin.30072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37924.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30072.log.gz 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39506.log.gz 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33899.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37924.log.gz 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37215.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38024.log 2024-09-04T21:57:15.713 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33899.log.gz 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34199.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37215.log.gz 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31967.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38024.log.gz/var/log/ceph/ceph-client.admin.34199.log: 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34199.log.gz -- 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/ceph-mon.c.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31967.log.gz 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38102.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38206.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38102.log.gz 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38206.log.gz 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2024-09-04T21:57:15.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30229.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33854.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38604.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30229.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33854.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32699.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35236.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38604.log: /var/log/ceph/ceph-client.admin.32699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38604.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32699.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33301.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30770.log 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35236.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33301.log.gz 2024-09-04T21:57:15.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38615.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30770.log.gz 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30753.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35906.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38615.log: /var/log/ceph/ceph-client.admin.30753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38615.log.gz 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30753.log.gz 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34798.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35906.log.gz 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34798.log.gz 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30359.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.audit.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31523.log 2024-09-04T21:57:15.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34149.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30359.log: /var/log/ceph/ceph-client.admin.31523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30359.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31523.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35491.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35012.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34149.log: /var/log/ceph/ceph-client.admin.35491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34149.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35491.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37722.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35012.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30159.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38258.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37722.log: /var/log/ceph/ceph-client.admin.30159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37722.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30159.log.gz 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32868.log 2024-09-04T21:57:15.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38258.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31992.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32868.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31992.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36192.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38362.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39402.log: /var/log/ceph/ceph-client.admin.36192.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36192.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33924.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38362.log: /var/log/ceph/ceph-client.admin.33924.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38362.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33924.log.gz 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35261.log 2024-09-04T21:57:15.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36895.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38815.log: /var/log/ceph/ceph-client.admin.35261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35261.log.gz 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32860.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36895.log.gz 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39480.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.32695.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32860.log.gz 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33040.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32695.log.gz 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33040.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38823.log 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-04T21:57:15.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz 2024-09-04T21:57:15.731 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35777.log 2024-09-04T21:57:15.731 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39350.log 2024-09-04T21:57:15.731 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38823.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36800.log 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35777.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39350.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38154.log 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32753.log 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36800.log: /var/log/ceph/ceph-client.admin.38154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36800.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38154.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33639.log 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37027.log 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32753.log: /var/log/ceph/ceph-client.admin.33639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32753.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33639.log.gz 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26426.log 2024-09-04T21:57:15.732 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35689.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37027.log: /var/log/ceph/ceph-client.admin.26426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37027.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26426.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30209.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38399.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35689.log: /var/log/ceph/ceph-client.admin.30209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35689.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30209.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32514.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37357.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38399.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32514.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32514.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37627.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37357.log.gz 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39194.log 2024-09-04T21:57:15.733 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37165.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37627.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39194.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31498.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39194.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37165.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36160.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38232.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.31498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31498.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36160.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33720.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36160.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38232.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34018.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.33720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33720.log.gz 2024-09-04T21:57:15.734 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34018.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34018.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30184.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36775.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36710.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30184.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36775.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37998.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36775.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.36710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36710.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34088.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37998.log: /var/log/ceph/ceph-client.admin.34088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37998.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34088.log.gz 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32436.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30307.log 2024-09-04T21:57:15.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2024-09-04T21:57:15.736 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.32436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32436.log.gz 2024-09-04T21:57:15.736 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37972.log 2024-09-04T21:57:15.736 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.30307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30307.log.gz 2024-09-04T21:57:15.736 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.37972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37972.log.gz 2024-09-04T21:57:15.844 INFO:teuthology.orchestra.run.smithi157.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-04T21:57:15.845 INFO:teuthology.orchestra.run.smithi112.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-04T21:57:15.989 INFO:teuthology.orchestra.run.smithi112.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-04T21:57:16.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-04T21:57:16.189 INFO:teuthology.orchestra.run.smithi112.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-04T21:57:16.642 INFO:teuthology.orchestra.run.smithi157.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-04T21:57:16.935 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-04T21:57:16.978 INFO:teuthology.orchestra.run.smithi157.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-04T21:57:17.014 INFO:teuthology.orchestra.run.smithi112.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-04T21:57:19.690 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-04T21:57:19.691 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-04T21:57:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m4.123s 2024-09-04T21:57:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m7.845s 2024-09-04T21:57:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.348s 2024-09-04T21:57:20.108 INFO:teuthology.orchestra.run.smithi157.stderr: 95.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-04T21:57:20.204 INFO:teuthology.orchestra.run.smithi157.stderr: 95.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-04T21:57:20.205 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-04T21:57:20.205 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m4.591s 2024-09-04T21:57:20.205 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m11.179s 2024-09-04T21:57:20.205 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m0.443s 2024-09-04T21:57:20.206 INFO:tasks.ceph:Archiving logs... 2024-09-04T21:57:20.207 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/remote/smithi112/log 2024-09-04T21:57:20.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-04T21:57:21.175 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/remote/smithi157/log 2024-09-04T21:57:21.175 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-04T21:57:22.497 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-04T21:57:22.506 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-04T21:57:22.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-04T21:57:22.509 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-04T21:57:22.588 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-04T21:57:22.598 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-04T21:57:22.598 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-04T21:57:22.600 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-04T21:57:22.612 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-04T21:57:22.612 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-04T21:57:22.612 INFO:teuthology.orchestra.run.smithi112.stdout:+hv01.front.sepi 67.205.162.81 3 u 1 64 177 0.085 -0.640 1.291 2024-09-04T21:57:22.612 INFO:teuthology.orchestra.run.smithi112.stdout:*hv02.front.sepi 63.231.80.2 3 u 4 64 177 0.110 -0.736 1.290 2024-09-04T21:57:22.612 INFO:teuthology.orchestra.run.smithi112.stdout:+hv03.front.sepi 74.6.168.72 3 u 2 64 177 0.102 -0.920 1.252 2024-09-04T21:57:22.612 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:57:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2024-09-04T21:57:22.614 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2024-09-04T21:57:22.614 INFO:teuthology.orchestra.run.smithi157.stdout:+hv01.front.sepi 67.205.162.81 3 u 1 64 177 0.062 -0.815 0.584 2024-09-04T21:57:22.614 INFO:teuthology.orchestra.run.smithi157.stdout:*hv02.front.sepi 63.231.80.2 3 u 2 64 177 0.137 -0.475 0.756 2024-09-04T21:57:22.614 INFO:teuthology.orchestra.run.smithi157.stdout:+hv03.front.sepi 74.6.168.72 3 u - 64 177 0.086 -0.960 0.601 2024-09-04T21:57:22.614 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T21:57:22.615 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-04T21:57:22.692 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-04T21:57:22.692 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-04T21:57:22.792 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-04T21:57:22.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-04T21:57:22.809 INFO:teuthology.task.internal:Duration was 850.369053 seconds 2024-09-04T21:57:22.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-04T21:57:22.819 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-04T21:57:22.819 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-04T21:57:22.821 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-04T21:57:22.858 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-04T21:57:22.858 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-09-04T21:57:22.859 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-04T21:57:22.910 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-09-04T21:57:22.910 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-04T21:57:22.920 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-04T21:57:22.920 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-04T21:57:22.959 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-04T21:57:22.991 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-04T21:57:22.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-04T21:57:23.031 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-04T21:57:23.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-04T21:57:23.116 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-04T21:57:23.117 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-04T21:57:23.127 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-04T21:57:23.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-04T21:57:23.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-04T21:57:23.171 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-04T21:57:23.178 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-09-04T21:57:23.188 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-09-04T21:57:23.220 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-04T21:57:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:57:23.249 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-04T21:57:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T21:57:23.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-04T21:57:23.280 INFO:teuthology.task.internal:Transferring archived files... 2024-09-04T21:57:23.281 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/remote/smithi112 2024-09-04T21:57:23.281 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-04T21:57:23.319 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-04_17:03:08-rbd-wip-dis-testing-distro-default-smithi/7889472/remote/smithi157 2024-09-04T21:57:23.319 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-04T21:57:23.341 INFO:teuthology.task.internal:Removing archive directory... 2024-09-04T21:57:23.342 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-04T21:57:23.355 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-04T21:57:23.383 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-04T21:57:23.396 INFO:teuthology.task.internal:Not uploading archives. 2024-09-04T21:57:23.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-04T21:57:23.405 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-04T21:57:23.406 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-04T21:57:23.408 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-04T21:57:23.412 INFO:teuthology.orchestra.run.smithi112.stdout: 658622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 4 21:57 /home/ubuntu/cephtest 2024-09-04T21:57:23.432 INFO:teuthology.orchestra.run.smithi157.stdout: 658636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 4 21:57 /home/ubuntu/cephtest 2024-09-04T21:57:23.434 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-04T21:57:23.445 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-04T21:57:23.468 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache} duration: 850.3690531253815 flavor: default owner: scheduled_dis@teuthology success: true 2024-09-04T21:57:23.468 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-04T21:57:23.546 INFO:teuthology.run:pass